Nov 02 20:12:01.863592 ip-10-0-134-206 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 20:12:01.863605 ip-10-0-134-206 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 20:12:01.863614 ip-10-0-134-206 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 20:12:01.863949 ip-10-0-134-206 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 20:12:12.112319 ip-10-0-134-206 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 20:12:12.112339 ip-10-0-134-206 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7f13762007964c4da9939a37d09f866d -- Nov 02 20:14:15.575941 ip-10-0-134-206 systemd[1]: Starting Kubernetes Kubelet... Nov 02 20:14:16.042511 ip-10-0-134-206 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:14:16.042511 ip-10-0-134-206 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 20:14:16.042511 ip-10-0-134-206 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:14:16.042511 ip-10-0-134-206 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:14:16.047634 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047558 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:16.047634 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047634 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:16.047634 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047637 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047641 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047644 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047647 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047649 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047653 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047655 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047658 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047660 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047663 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047666 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047682 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047686 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047689 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047691 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047694 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047697 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047699 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047702 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047704 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:16.047740 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047707 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047709 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047712 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047715 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047718 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047720 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047723 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047727 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047732 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047735 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047738 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047741 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047744 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047747 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047750 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047752 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047755 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047758 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047760 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:16.048197 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047763 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047766 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047768 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047771 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047774 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047777 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047780 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047782 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047785 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047787 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047790 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047794 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047797 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047801 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047803 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047806 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047809 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047812 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047814 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:16.048640 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047817 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047820 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047822 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047825 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047827 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047830 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047833 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047835 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047838 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047840 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047843 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047845 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047854 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047857 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047859 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047862 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047864 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047867 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047869 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:16.049118 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047872 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047875 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047878 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047881 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047883 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047886 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047888 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047890 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047893 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047896 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047899 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047902 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047904 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047907 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047910 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047913 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047915 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047918 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047920 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047923 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:16.049558 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047926 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047928 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047931 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047933 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047936 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047938 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.047941 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048710 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048718 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048721 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048724 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048727 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048729 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048731 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048735 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048738 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048741 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048743 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048746 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048748 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:16.050031 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048751 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048754 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048756 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048759 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048762 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048764 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048768 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048770 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048773 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048775 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048778 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048780 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048783 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048785 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048788 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048790 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048793 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048795 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048798 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:16.050485 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048800 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048802 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048805 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048807 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048809 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048812 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048814 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048817 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048820 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048822 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048825 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048827 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048829 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048832 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048835 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048837 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048840 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048842 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:16.050998 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048845 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048853 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048857 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048860 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048863 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048866 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048868 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048871 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048874 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048876 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048879 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048882 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048884 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048886 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048889 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048892 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048894 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048897 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048899 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:16.051523 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048902 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048904 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048907 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048909 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048912 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048915 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048917 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048919 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048922 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048925 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048928 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048931 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048933 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048936 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048940 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048943 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048945 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048948 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048950 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048953 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:16.051995 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048955 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048958 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048961 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048964 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048968 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048970 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048973 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048975 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048978 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048981 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048983 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048985 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048988 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048990 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048994 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048997 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.048999 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049506 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049512 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:16.052463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049514 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049519 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049523 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049526 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049529 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049532 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049534 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049539 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049541 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049544 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049547 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049549 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049552 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049554 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049557 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049559 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049562 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049565 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049567 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049570 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:16.052929 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049573 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049575 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049578 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049580 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049583 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049585 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049588 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049590 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049593 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049595 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049598 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049601 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049603 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049606 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049609 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049611 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049614 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049616 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049619 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:16.053396 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049622 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049624 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049627 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049630 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049632 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049634 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049637 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049639 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049642 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049644 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049646 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049649 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049651 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049653 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049656 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049658 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049661 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049663 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049665 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049679 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:16.053857 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049682 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049684 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049687 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049689 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049695 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049697 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049700 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049703 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049705 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049709 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049711 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049714 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049716 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049719 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049721 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049724 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049726 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049729 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049732 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:16.054322 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049734 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049736 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049739 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049741 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049744 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049746 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049749 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049751 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049755 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049759 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049762 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049765 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049768 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049770 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049773 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049775 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049778 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049780 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049785 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:16.054769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049789 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049791 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049794 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049796 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049799 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049802 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.049804 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050907 2569 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050916 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050923 2569 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050928 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050932 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050936 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050944 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050948 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050951 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050956 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050961 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050966 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050970 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050974 2569 flags.go:64] FLAG: --cgroup-root="" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050977 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050981 2569 flags.go:64] FLAG: --client-ca-file="" Nov 02 20:14:16.055196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050985 2569 flags.go:64] FLAG: --cloud-provider="external" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050989 2569 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.050995 2569 flags.go:64] FLAG: --cluster-domain="" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051000 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051004 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051009 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051014 2569 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051020 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051025 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051033 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051039 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051044 2569 flags.go:64] FLAG: --contention-profiling="false" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051049 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051053 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051058 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051063 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051068 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051071 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051074 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051077 2569 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051080 2569 flags.go:64] FLAG: --enable-server="true" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051083 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051088 2569 flags.go:64] FLAG: --event-burst="100" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051091 2569 flags.go:64] FLAG: --event-qps="50" Nov 02 20:14:16.055733 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051094 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051097 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051100 2569 flags.go:64] FLAG: --eviction-hard="" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051104 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051108 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051111 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051113 2569 flags.go:64] FLAG: --eviction-soft="" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051116 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051119 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051123 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051127 2569 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051130 2569 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051133 2569 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051136 2569 flags.go:64] FLAG: --feature-gates="" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051140 2569 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051143 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051146 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051150 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051154 2569 flags.go:64] FLAG: --healthz-port="10248" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051157 2569 flags.go:64] FLAG: --help="false" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051160 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051163 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051166 2569 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 20:14:16.056345 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051170 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051173 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051177 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051180 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051182 2569 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051185 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051188 2569 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051191 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051194 2569 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051197 2569 flags.go:64] FLAG: --kube-reserved="" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051200 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051202 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051205 2569 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051208 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051211 2569 flags.go:64] FLAG: --lock-file="" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051214 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051217 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051220 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051225 2569 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051230 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051233 2569 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051236 2569 flags.go:64] FLAG: --logging-format="text" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051240 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051243 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 20:14:16.056898 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051246 2569 flags.go:64] FLAG: --manifest-url="" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051249 2569 flags.go:64] FLAG: --manifest-url-header="" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051253 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051257 2569 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051261 2569 flags.go:64] FLAG: --max-pods="110" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051264 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051267 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051270 2569 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051273 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051276 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051278 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051281 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051288 2569 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051291 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051294 2569 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051298 2569 flags.go:64] FLAG: --pod-cidr="" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051301 2569 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051304 2569 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051307 2569 flags.go:64] FLAG: --pods-per-core="0" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051310 2569 flags.go:64] FLAG: --port="10250" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051313 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051316 2569 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c184f5e24f5baa55" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051319 2569 flags.go:64] FLAG: --qos-reserved="" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051322 2569 flags.go:64] FLAG: --read-only-port="10255" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051324 2569 flags.go:64] FLAG: --register-node="true" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051327 2569 flags.go:64] FLAG: --register-with-taints="" Nov 02 20:14:16.057450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051331 2569 flags.go:64] FLAG: --registry-burst="10" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051335 2569 flags.go:64] FLAG: --registry-qps="5" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051338 2569 flags.go:64] FLAG: --reserved-cpus="" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051341 2569 flags.go:64] FLAG: --reserved-memory="" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051345 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051348 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051351 2569 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051354 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051358 2569 flags.go:64] FLAG: --runonce="false" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051361 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051364 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051368 2569 flags.go:64] FLAG: --seccomp-default="false" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051372 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051375 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051378 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051381 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051384 2569 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051387 2569 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051390 2569 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051393 2569 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051395 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051398 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051401 2569 flags.go:64] FLAG: --system-cgroups="" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051404 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051409 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 20:14:16.058220 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051412 2569 flags.go:64] FLAG: --tls-cert-file="" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051415 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051419 2569 flags.go:64] FLAG: --tls-min-version="" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051422 2569 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051425 2569 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051428 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051430 2569 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051433 2569 flags.go:64] FLAG: --v="2" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051440 2569 flags.go:64] FLAG: --version="false" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051444 2569 flags.go:64] FLAG: --vmodule="" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051448 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051451 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051614 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051618 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051622 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051626 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051628 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051631 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051633 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051639 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051642 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051644 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:16.059083 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051646 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051650 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051652 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051655 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051657 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051659 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051662 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051665 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051681 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051684 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051686 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051689 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051692 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051695 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051697 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051700 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051702 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051705 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051709 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:16.059605 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051712 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051714 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051717 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051719 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051722 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051724 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051728 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051731 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051733 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051736 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051738 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051741 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051744 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051747 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051750 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051754 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051757 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051760 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051763 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051765 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:16.060060 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051768 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051770 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051773 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051776 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051779 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051781 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051784 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051786 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051789 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051791 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051793 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051798 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051800 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051803 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051805 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051808 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051811 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051813 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051817 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:16.060554 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051819 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051822 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051826 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051829 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051831 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051834 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051837 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051840 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051842 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051845 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051854 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051857 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051859 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051862 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051864 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051867 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051869 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051872 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051874 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051877 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:16.061016 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051879 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051882 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051885 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051887 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051891 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051893 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051896 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051898 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051901 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051903 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051906 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051910 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051912 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051915 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051918 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051920 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051923 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:16.061476 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051925 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.051954 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051991 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051995 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.051998 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052001 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052004 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052007 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052010 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052013 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052016 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052018 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052021 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052023 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052025 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052028 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:16.061887 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052030 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052033 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052035 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052037 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052042 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052044 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052047 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052050 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052052 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052055 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052059 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052061 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052064 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052066 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052069 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052071 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052074 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052076 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052079 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052082 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:16.062251 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052085 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052088 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052090 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052093 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052095 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052097 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052100 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052102 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052105 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052108 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052110 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052113 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052115 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052118 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052120 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052123 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052127 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052129 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052132 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:16.062714 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052134 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052138 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052140 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052144 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052147 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052149 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052152 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052154 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052157 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052159 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052162 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052165 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052168 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052170 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052175 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052178 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052181 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052183 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052186 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052188 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:16.063142 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052191 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052194 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052196 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052199 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052202 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052204 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052207 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052209 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052211 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052215 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052218 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052220 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052223 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052225 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052229 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052233 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052236 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052238 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052241 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:16.063599 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052244 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052247 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052251 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052254 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052256 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052259 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052262 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052264 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052267 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052270 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052273 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052275 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052278 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.052280 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.060379 2569 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.060396 2569 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.060412 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.060418 2569 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060481 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:16.064065 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060486 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060490 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060493 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060496 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060499 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060525 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060531 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060535 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060538 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060541 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060544 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060546 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060549 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060551 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060554 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060557 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060559 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060562 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060565 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:16.064503 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060568 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060571 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060574 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060576 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060579 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060581 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060584 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060588 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060593 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060597 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060601 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060604 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060607 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060610 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060612 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060615 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060618 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060620 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060623 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:16.065564 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060625 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060627 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060630 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060632 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060635 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060639 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060643 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060648 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060651 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060654 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060657 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060660 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060662 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060665 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060680 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060685 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060688 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060690 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:16.066020 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060693 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060695 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060698 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060700 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060703 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060705 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060708 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060710 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060713 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060715 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060717 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060720 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060723 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060726 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060728 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060731 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060734 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060737 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060739 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060742 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:16.066463 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060744 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060747 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060751 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060756 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060760 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060764 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060768 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060771 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060773 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060775 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060778 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060780 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060783 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060785 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060788 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060790 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060792 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060795 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060797 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060800 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:16.066915 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060802 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060805 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060807 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060810 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060812 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060815 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060817 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060820 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060822 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.060854 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.060993 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061000 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061004 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061007 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061010 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061014 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:16.067351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061017 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061019 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061022 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061025 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061027 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061030 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061032 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061035 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061038 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061040 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061043 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061045 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061048 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061050 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061053 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061055 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061057 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061060 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061062 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:16.067719 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061065 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061069 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061073 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061077 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061081 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061086 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061090 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061094 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061096 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061099 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061102 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061104 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061107 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061110 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061113 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061115 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061117 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061120 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061122 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:16.068158 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061125 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061127 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061130 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061132 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061134 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061137 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061139 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061142 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061144 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061147 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061151 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061155 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061160 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061163 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061166 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061168 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061171 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061174 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061178 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061181 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:16.068607 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061183 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061186 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061188 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061190 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061193 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061195 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061197 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061201 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061204 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061206 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061209 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061211 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061213 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061217 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061220 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061224 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061228 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061234 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061239 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:16.069070 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061243 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061246 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061249 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061251 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061254 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061256 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061259 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061262 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061264 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061267 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061269 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061274 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061277 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061280 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061283 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061285 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061287 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061290 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061292 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:16.069576 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061295 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061298 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061301 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061304 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.061332 2569 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061367 2569 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061371 2569 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061373 2569 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061377 2569 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061380 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061384 2569 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061389 2569 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061394 2569 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061399 2569 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061403 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:14:16.070029 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061406 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061409 2569 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061411 2569 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061414 2569 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061416 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061419 2569 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061421 2569 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061424 2569 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061426 2569 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061431 2569 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061433 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061436 2569 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061438 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061441 2569 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061443 2569 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061445 2569 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061448 2569 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061450 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061453 2569 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061455 2569 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061458 2569 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:14:16.070422 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061461 2569 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061464 2569 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061467 2569 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061471 2569 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061479 2569 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061483 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061486 2569 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061489 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061491 2569 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061494 2569 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061496 2569 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061499 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061501 2569 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061503 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061506 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061508 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061511 2569 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061513 2569 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061516 2569 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061518 2569 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:14:16.070970 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061522 2569 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061524 2569 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061528 2569 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061531 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061534 2569 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061537 2569 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061539 2569 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061542 2569 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061544 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061547 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061552 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061556 2569 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061561 2569 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061565 2569 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061568 2569 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061570 2569 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061574 2569 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061577 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061580 2569 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:14:16.071434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061583 2569 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061586 2569 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061588 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061591 2569 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061593 2569 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061596 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061598 2569 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061601 2569 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061603 2569 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061606 2569 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061608 2569 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061611 2569 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061613 2569 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061617 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061619 2569 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061621 2569 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061624 2569 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061627 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061632 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061636 2569 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:14:16.071909 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061640 2569 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061644 2569 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061646 2569 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061649 2569 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061652 2569 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061655 2569 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061658 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061661 2569 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061663 2569 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061677 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061680 2569 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061682 2569 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061685 2569 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061687 2569 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061690 2569 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:16.061692 2569 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.061756 2569 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.065085 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.066111 2569 server.go:1014] "Starting client certificate rotation" Nov 02 20:14:16.072351 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.066206 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 20:14:16.072818 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.066242 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 20:14:16.072818 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.068268 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:14:16.072818 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.069420 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:14:16.081076 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.081059 2569 log.go:25] "Validated CRI v1 runtime API" Nov 02 20:14:16.085295 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.085280 2569 log.go:25] "Validated CRI v1 image API" Nov 02 20:14:16.086099 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.086074 2569 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 20:14:16.088686 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.088657 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9f7e624d-14aa-49af-97ae-565ab1507b7a:/dev/nvme0n1p3 dfce8e62-9211-47ba-b230-100974542e48:/dev/nvme0n1p4] Nov 02 20:14:16.088743 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.088691 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 20:14:16.093897 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.093790 2569 manager.go:217] Machine: {Timestamp:2026-11-02 20:14:16.09209567 +0000 UTC m=+0.380872604 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200994 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e45c1b6f838d062e5f90aadd8200 SystemUUID:ec26e45c-1b6f-838d-062e-5f90aadd8200 BootID:7f137620-0796-4c4d-a993-9a37d09f866d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:df:bb:71:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:df:bb:71:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:71:90:e3:3f:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 20:14:16.094563 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.094554 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 20:14:16.094641 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.094630 2569 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 20:14:16.098452 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.098272 2569 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 20:14:16.098589 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.098454 2569 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-206.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 20:14:16.098643 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.098592 2569 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 20:14:16.098643 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.098601 2569 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 20:14:16.098643 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.098629 2569 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 20:14:16.098728 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.098644 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 20:14:16.099730 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.099720 2569 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 20:14:16.104057 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.104045 2569 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 20:14:16.104182 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.104173 2569 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 20:14:16.105001 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.104992 2569 kubelet.go:493] "Attempting to sync node with API server" Nov 02 20:14:16.105053 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.105005 2569 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 20:14:16.105053 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.105017 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 20:14:16.105053 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.105026 2569 kubelet.go:405] "Adding apiserver pod source" Nov 02 20:14:16.105053 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.105034 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 20:14:16.105321 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.105307 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 20:14:16.107863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.107842 2569 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 20:14:16.113872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.113858 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 20:14:16.114789 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.114776 2569 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 20:14:16.114851 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.114814 2569 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 20:14:16.114996 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.114983 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 20:14:16.115061 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115003 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 20:14:16.115061 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115013 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 20:14:16.115061 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115022 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 20:14:16.115061 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115030 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 20:14:16.115061 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115040 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 20:14:16.115746 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115736 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 20:14:16.115794 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115749 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 20:14:16.115794 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115760 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 20:14:16.115794 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115770 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 20:14:16.115794 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115784 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 20:14:16.115975 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.115799 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 20:14:16.116453 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.116442 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 20:14:16.116501 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.116458 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 20:14:16.120206 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.120190 2569 server.go:1263] "Started kubelet" Nov 02 20:14:16.120285 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.120235 2569 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 20:14:16.120361 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.120308 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 20:14:16.120405 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.120397 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 20:14:16.120854 ip-10-0-134-206 systemd[1]: Started Kubernetes Kubelet. Nov 02 20:14:16.122029 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.122007 2569 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 20:14:16.123074 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.123062 2569 server.go:317] "Adding debug handlers to kubelet server" Nov 02 20:14:16.125704 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.125659 2569 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 20:14:16.128931 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.128910 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 20:14:16.130284 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.130271 2569 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 20:14:16.130409 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.130391 2569 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-206.us-west-2.compute.internal\" not found" Nov 02 20:14:16.131049 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131032 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 20:14:16.131132 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131056 2569 factory.go:55] Registering systemd factory Nov 02 20:14:16.131132 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131062 2569 factory.go:223] Registration of the systemd container factory successfully Nov 02 20:14:16.131231 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131198 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 20:14:16.131285 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131271 2569 factory.go:153] Registering CRI-O factory Nov 02 20:14:16.131315 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131289 2569 factory.go:223] Registration of the crio container factory successfully Nov 02 20:14:16.131315 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131309 2569 factory.go:103] Registering Raw factory Nov 02 20:14:16.131367 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131321 2569 manager.go:1199] Started watching for new ooms in manager Nov 02 20:14:16.131423 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131196 2569 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 20:14:16.131461 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131425 2569 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 20:14:16.131513 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131499 2569 reconstruct.go:97] "Volume reconstruction finished" Nov 02 20:14:16.131513 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.131510 2569 reconciler.go:29] "Reconciler: start to sync state" Nov 02 20:14:16.132347 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.132332 2569 manager.go:319] Starting recovery of all containers Nov 02 20:14:16.144463 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.144445 2569 manager.go:324] Recovery completed Nov 02 20:14:16.148481 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.148470 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:16.150319 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.150302 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:16.150385 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.150339 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:16.150385 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.150357 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:16.150869 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.150854 2569 cpu_manager.go:226] "Starting" policy="none" Nov 02 20:14:16.150932 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.150868 2569 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 20:14:16.150932 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.150889 2569 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 20:14:16.150932 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.150924 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:16.151554 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.151538 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:16.151649 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.151565 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:16.151649 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.151579 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:16.153149 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.153137 2569 policy_none.go:50] "Start" Nov 02 20:14:16.153197 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.153153 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 20:14:16.153197 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.153168 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 20:14:16.155153 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.155137 2569 policy_none.go:44] "Start" Nov 02 20:14:16.198250 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.198191 2569 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 20:14:16.200816 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.200802 2569 manager.go:341] "Starting Device Plugin manager" Nov 02 20:14:16.201760 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.201743 2569 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 20:14:16.201760 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.201757 2569 server.go:87] "Starting device plugin registration server" Nov 02 20:14:16.201963 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.201953 2569 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 20:14:16.202004 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.201964 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 20:14:16.202068 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.202049 2569 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 20:14:16.202148 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.202134 2569 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 20:14:16.202148 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.202147 2569 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 20:14:16.202658 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.202638 2569 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 20:14:16.202744 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.202705 2569 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-206.us-west-2.compute.internal\" not found" Nov 02 20:14:16.258586 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.258544 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-206.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 20:14:16.258586 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.258542 2569 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-206.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 20:14:16.260533 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.258556 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-206.us-west-2.compute.internal.18e4539630428374 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-206.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-206.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 20:14:16.120165236 +0000 UTC m=+0.408942169,LastTimestamp:2026-11-02 20:14:16.120165236 +0000 UTC m=+0.408942169,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-206.us-west-2.compute.internal,}" Nov 02 20:14:16.263103 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.263074 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 20:14:16.264266 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.264244 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 20:14:16.264356 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.264269 2569 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 20:14:16.264356 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.264291 2569 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 20:14:16.264356 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.264330 2569 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 20:14:16.302353 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.302309 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:16.303130 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.303109 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:16.303213 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.303138 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:16.303213 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.303148 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:16.303213 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.303171 2569 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.351133 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.350974 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 20:14:16.351133 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.351067 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pk4tr" Nov 02 20:14:16.365391 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.365367 2569 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal"] Nov 02 20:14:16.365487 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.365436 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:16.366316 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.366302 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:16.366396 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.366327 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:16.366396 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.366337 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:16.368529 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.368515 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:16.368687 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.368659 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.368757 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.368732 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:16.369282 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.369267 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:16.369363 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.369298 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:16.369363 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.369312 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:16.369460 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.369312 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:16.369460 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.369397 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:16.369460 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.369410 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:16.371494 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.371479 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.371572 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.371502 2569 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:14:16.372554 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.372534 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:14:16.372632 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.372570 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:14:16.372632 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.372588 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:14:16.390071 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.390051 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-206.us-west-2.compute.internal\" not found" node="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.391978 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.391963 2569 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-206.us-west-2.compute.internal\" not found" node="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.414178 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.414152 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:16.418584 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.418562 2569 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:16.433468 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.433443 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d51646e0acf14788341d5bd3c94d4017-config\") pod \"kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"d51646e0acf14788341d5bd3c94d4017\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.433551 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.433475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.433551 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.433500 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.507209 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.507185 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:16.508442 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.508424 2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:16.508677 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.508657 2569 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.508780 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:16.508697 2569 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-206.us-west-2.compute.internal\": node \"ip-10-0-134-206.us-west-2.compute.internal\" not found" Nov 02 20:14:16.531211 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.531191 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.534443 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.534420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.534523 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.534450 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.534523 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.534467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d51646e0acf14788341d5bd3c94d4017-config\") pod \"kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"d51646e0acf14788341d5bd3c94d4017\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.534586 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.534529 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.534586 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.534580 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d51646e0acf14788341d5bd3c94d4017-config\") pod \"kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"d51646e0acf14788341d5bd3c94d4017\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.534649 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.534616 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.593639 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.593576 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:14:16.593745 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.593685 2569 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.648963 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.648948 2569 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:14:16.695358 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.695334 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:16.697464 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:16.697449 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:14:17.065897 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.065804 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 20:14:17.066426 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.066002 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 20:14:17.066426 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.066012 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 20:14:17.066426 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.066024 2569 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 20:14:17.066426 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.066006 2569 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a005d902b89f44e8db632ef0d2b795eb-22154488e9922e7d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-206.us-west-2.compute.internal.18e45396320ea56e\": read tcp 10.0.134.206:58040->18.190.165.24:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-206.us-west-2.compute.internal.18e45396320ea56e default 5235 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-206.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-206.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-206.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 20:14:16 +0000 UTC,LastTimestamp:2026-11-02 20:14:16.366314842 +0000 UTC m=+0.655091776,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-206.us-west-2.compute.internal,}" Nov 02 20:14:17.105758 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.105726 2569 apiserver.go:52] "Watching apiserver" Nov 02 20:14:17.130072 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.130043 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 20:14:17.357186 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.357089 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 20:09:16 +0000 UTC" deadline="2028-05-07 19:13:51.569666561 +0000 UTC" Nov 02 20:14:17.357186 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.357124 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13246h59m34.212546167s" Nov 02 20:14:17.380517 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.380488 2569 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 20:14:17.380890 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.380869 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-4lc2l","openshift-multus/multus-additional-cni-plugins-6ks29","openshift-network-diagnostics/network-check-target-89w6h","openshift-multus/network-metrics-daemon-ndz76","kube-system/global-pull-secret-syncer-s49d6","openshift-image-registry/node-ca-8hm6m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-2l8lk","openshift-ovn-kubernetes/ovnkube-node-8plx8","kube-system/konnectivity-agent-wnw5s","kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr","openshift-cluster-node-tuning-operator/tuned-898kr","openshift-dns/node-resolver-6zxch"] Nov 02 20:14:17.383567 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.383545 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.385886 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.385861 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.388200 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.388184 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.390448 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.390433 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.390522 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.390503 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:17.392704 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.392688 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.394979 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.394962 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:17.395068 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.395048 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:17.397324 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.397305 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.399769 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.399747 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:17.399907 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.399883 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:17.402222 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.402205 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:17.406382 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.406356 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.408834 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.408815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.410993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.410977 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.432358 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.432341 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 20:14:17.436510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.436483 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-t6dvk\"" Nov 02 20:14:17.436510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.436489 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 20:14:17.436637 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.436512 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 20:14:17.436637 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.436530 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 20:14:17.436637 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.436540 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 20:14:17.437919 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.437891 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 20:14:17.438004 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.437979 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxslf\"" Nov 02 20:14:17.438004 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.437983 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 20:14:17.440081 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440055 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 20:14:17.440081 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440060 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25wg\" (UniqueName: \"kubernetes.io/projected/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-kube-api-access-n25wg\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.440234 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-tuned\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.440234 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2dabbc-40dc-4d65-b445-bcde680092ad-hosts-file\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.440234 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-env-overrides\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.440234 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8rww\" (UniqueName: \"kubernetes.io/projected/2a85e903-5cfa-4bb0-b902-d510337216c9-kube-api-access-x8rww\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.440437 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cni-binary-copy\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.440437 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-conf\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.440437 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-systemd\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.440437 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440331 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-lib-modules\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.440437 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440347 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-dbus\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.440437 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-registration-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.440437 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440425 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-slash\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.440437 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440441 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-etc-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440457 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-k8s-cni-cncf-io\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-daemon-config\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440506 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440526 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440542 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440553 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440584 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2dabbc-40dc-4d65-b445-bcde680092ad-tmp-dir\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440593 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440624 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-kubelet\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440639 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440663 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-config\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440706 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a85e903-5cfa-4bb0-b902-d510337216c9-ovn-node-metrics-cert\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440720 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7m5tr\"" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440731 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-modprobe-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-host\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-serviceca\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/168a3d5f-16fa-44c2-8123-bcf76bdad253-host-slash\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbsdl\" (UniqueName: \"kubernetes.io/projected/168a3d5f-16fa-44c2-8123-bcf76bdad253-kube-api-access-wbsdl\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440836 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-conf-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.440872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440853 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-script-lib\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440933 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-cnibin\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.440983 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ae28258-51a5-4e33-8040-39d8592006b8-agent-certs\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4pn\" (UniqueName: \"kubernetes.io/projected/1f2dabbc-40dc-4d65-b445-bcde680092ad-kube-api-access-jd4pn\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441029 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zj4q\" (UniqueName: \"kubernetes.io/projected/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-kube-api-access-4zj4q\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441046 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbxn\" (UniqueName: \"kubernetes.io/projected/1d5f7eda-1c08-4147-9fc8-241e0c982451-kube-api-access-cwbxn\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441066 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-multus-certs\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441089 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vzml\" (UniqueName: \"kubernetes.io/projected/9a4c7d5d-9276-4dce-8e0a-d7782b067792-kube-api-access-8vzml\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441105 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-tmp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-bin\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441140 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441154 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-netns\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-node-log\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441197 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.441863 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441226 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-kubelet-config\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441253 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-device-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-systemd-units\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-netd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441378 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysconfig\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441392 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-kubernetes\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-bin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441438 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441457 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-socket-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441479 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-system-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441502 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-multus\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441523 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-kubelet\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441548 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441579 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-ovn\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441615 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/168a3d5f-16fa-44c2-8123-bcf76bdad253-iptables-alerter-script\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.442693 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441654 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441681 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-sys-fs\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-netns\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-etc-kubernetes\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441772 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-var-lib-kubelet\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441792 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-log-socket\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-run\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-socket-dir-parent\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-system-cni-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441866 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cvrw\" (UniqueName: \"kubernetes.io/projected/31c97247-cf8b-43b8-8fe4-4979ff070278-kube-api-access-7cvrw\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441888 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gqp\" (UniqueName: \"kubernetes.io/projected/eab012ae-9e57-45bb-8f50-9fe5f511a30c-kube-api-access-49gqp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441919 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-systemd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441936 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.441962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ae28258-51a5-4e33-8040-39d8592006b8-konnectivity-ca\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.442040 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-var-lib-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.443488 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.442062 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-sys\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.444342 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.442083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.444342 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.442098 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-etc-selinux\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.444342 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.442112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-hostroot\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.444342 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.442149 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.444342 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.442170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-host\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.444342 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.442197 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cnibin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.444888 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.444868 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 20:14:17.444954 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.444941 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vjrtf\"" Nov 02 20:14:17.444993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.444981 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 20:14:17.445025 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.444988 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 20:14:17.483443 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.483425 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:17.483532 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.483485 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qfmz2\"" Nov 02 20:14:17.486729 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.486717 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 20:14:17.486917 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.486900 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:17.487083 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.487062 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:14:17.487368 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.487134 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 20:14:17.487368 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.487164 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ldshw\"" Nov 02 20:14:17.487368 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.487172 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 20:14:17.490133 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490114 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 20:14:17.490476 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490457 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kz985\"" Nov 02 20:14:17.490572 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490457 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 20:14:17.490631 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490591 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 20:14:17.490704 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490650 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 20:14:17.490704 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490695 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5vs7j\"" Nov 02 20:14:17.490811 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490724 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 20:14:17.490811 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490783 2569 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 20:14:17.490892 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490815 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 20:14:17.490892 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490868 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xcmx8" Nov 02 20:14:17.490972 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.490957 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xxqx9\"" Nov 02 20:14:17.543166 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543147 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-bin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543274 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.543274 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-socket-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.543274 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-system-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543274 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543233 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-multus\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543274 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543248 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-bin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543274 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543267 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-kubelet\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543322 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-system-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543328 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543324 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-multus\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543330 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-kubelet\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-ovn\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543391 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-socket-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/168a3d5f-16fa-44c2-8123-bcf76bdad253-iptables-alerter-script\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543454 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-ovn\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.543510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543470 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543516 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-sys-fs\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-netns\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-etc-kubernetes\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543603 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543636 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-var-lib-kubelet\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543638 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-netns\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543666 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-sys-fs\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-log-socket\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-run\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543752 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-log-socket\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543759 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-socket-dir-parent\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-system-cni-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543728 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-etc-kubernetes\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cvrw\" (UniqueName: \"kubernetes.io/projected/31c97247-cf8b-43b8-8fe4-4979ff070278-kube-api-access-7cvrw\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543868 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-run\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543870 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-system-cni-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.543958 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543891 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-socket-dir-parent\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543940 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gqp\" (UniqueName: \"kubernetes.io/projected/eab012ae-9e57-45bb-8f50-9fe5f511a30c-kube-api-access-49gqp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543967 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-systemd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544013 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-systemd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544047 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/168a3d5f-16fa-44c2-8123-bcf76bdad253-iptables-alerter-script\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544058 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544158 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544205 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-var-lib-kubelet\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.543993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ae28258-51a5-4e33-8040-39d8592006b8-konnectivity-ca\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-var-lib-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544595 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-sys\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544632 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544645 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-var-lib-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.544655 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544687 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-sys\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.544809 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-etc-selinux\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.544696 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545044 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-etc-selinux\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-hostroot\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.545135 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs podName:9a4c7d5d-9276-4dce-8e0a-d7782b067792 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:18.045098719 +0000 UTC m=+2.333875659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs") pod "network-metrics-daemon-ndz76" (UID: "9a4c7d5d-9276-4dce-8e0a-d7782b067792") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545147 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-hostroot\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-host\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545273 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cnibin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25wg\" (UniqueName: \"kubernetes.io/projected/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-kube-api-access-n25wg\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-tuned\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2dabbc-40dc-4d65-b445-bcde680092ad-hosts-file\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-env-overrides\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545468 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ae28258-51a5-4e33-8040-39d8592006b8-konnectivity-ca\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:17.545490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rww\" (UniqueName: \"kubernetes.io/projected/2a85e903-5cfa-4bb0-b902-d510337216c9-kube-api-access-x8rww\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545510 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cni-binary-copy\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545541 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-conf\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-systemd\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-lib-modules\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545599 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-host\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-dbus\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545690 2569 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545948 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-dbus\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.546142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.545995 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2dabbc-40dc-4d65-b445-bcde680092ad-hosts-file\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546268 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cnibin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-registration-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546339 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-slash\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-etc-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546392 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-k8s-cni-cncf-io\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546424 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-daemon-config\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546460 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-conf\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2dabbc-40dc-4d65-b445-bcde680092ad-tmp-dir\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.546547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546516 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-slash\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546573 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-registration-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546587 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-lib-modules\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546616 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-kubelet\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-config\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546634 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-systemd\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546693 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-k8s-cni-cncf-io\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546747 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a85e903-5cfa-4bb0-b902-d510337216c9-ovn-node-metrics-cert\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-modprobe-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546796 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-etc-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546820 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-host\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546854 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-host\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546864 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-serviceca\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/168a3d5f-16fa-44c2-8123-bcf76bdad253-host-slash\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.546993 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.546974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbsdl\" (UniqueName: \"kubernetes.io/projected/168a3d5f-16fa-44c2-8123-bcf76bdad253-kube-api-access-wbsdl\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-conf-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547072 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-script-lib\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547108 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.547223 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547239 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-modprobe-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547312 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-kubelet\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.547305 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret podName:6b02c000-6d43-4adf-ba8b-4afd3e1186bc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:18.047274156 +0000 UTC m=+2.336051091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret") pod "global-pull-secret-syncer-s49d6" (UID: "6b02c000-6d43-4adf-ba8b-4afd3e1186bc") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547375 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-conf-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547414 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/168a3d5f-16fa-44c2-8123-bcf76bdad253-host-slash\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.547650 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547044 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cni-binary-copy\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.548127 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547727 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2dabbc-40dc-4d65-b445-bcde680092ad-tmp-dir\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.548127 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-cnibin\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.548127 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547769 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-env-overrides\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.548127 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547777 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.548127 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547818 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-cnibin\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.548127 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ae28258-51a5-4e33-8040-39d8592006b8-agent-certs\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:17.548127 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.547921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4pn\" (UniqueName: \"kubernetes.io/projected/1f2dabbc-40dc-4d65-b445-bcde680092ad-kube-api-access-jd4pn\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.548127 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548036 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.548490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548397 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-config\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.548490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548416 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zj4q\" (UniqueName: \"kubernetes.io/projected/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-kube-api-access-4zj4q\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.548490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548476 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbxn\" (UniqueName: \"kubernetes.io/projected/1d5f7eda-1c08-4147-9fc8-241e0c982451-kube-api-access-cwbxn\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.548618 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-multus-certs\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.548618 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548510 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-script-lib\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.548618 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548550 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzml\" (UniqueName: \"kubernetes.io/projected/9a4c7d5d-9276-4dce-8e0a-d7782b067792-kube-api-access-8vzml\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:17.548618 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-tmp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.548833 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-bin\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.548833 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.548833 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548760 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-netns\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.548833 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-node-log\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549006 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548822 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549006 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548862 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-kubelet-config\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.549006 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-device-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.549006 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.548957 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-systemd-units\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549182 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549011 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549182 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549057 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-netd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549182 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysconfig\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.549317 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-kubernetes\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.549364 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549329 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-kubernetes\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.549364 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549328 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-multus-certs\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.549455 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549375 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-device-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.549455 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549389 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-kubelet-config\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:17.549549 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549473 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-node-log\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549593 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549554 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-systemd-units\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549671 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-netns\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549718 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.549974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549726 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-bin\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549756 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-daemon-config\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.549974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549801 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549843 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-netd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549872 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.549974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.549935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysconfig\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.551349 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.551024 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-tuned\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.552143 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.552049 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.552143 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.552049 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.552143 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.552102 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-serviceca\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.552296 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.552211 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ae28258-51a5-4e33-8040-39d8592006b8-agent-certs\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:17.552296 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.552217 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-tmp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.553735 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.553717 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a85e903-5cfa-4bb0-b902-d510337216c9-ovn-node-metrics-cert\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.602880 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.602848 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:14:17.602880 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.602874 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:14:17.603035 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.602888 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8s82w for pod openshift-network-diagnostics/network-check-target-89w6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:17.603035 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:17.602954 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w podName:75ef20c6-f775-4fc4-96e3-8bd5f4a983f2 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:18.102936027 +0000 UTC m=+2.391712972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8s82w" (UniqueName: "kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w") pod "network-check-target-89w6h" (UID: "75ef20c6-f775-4fc4-96e3-8bd5f4a983f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:17.604970 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.604949 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gqp\" (UniqueName: \"kubernetes.io/projected/eab012ae-9e57-45bb-8f50-9fe5f511a30c-kube-api-access-49gqp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.605218 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.605195 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cvrw\" (UniqueName: \"kubernetes.io/projected/31c97247-cf8b-43b8-8fe4-4979ff070278-kube-api-access-7cvrw\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.606989 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.606962 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25wg\" (UniqueName: \"kubernetes.io/projected/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-kube-api-access-n25wg\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.608086 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.608027 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbsdl\" (UniqueName: \"kubernetes.io/projected/168a3d5f-16fa-44c2-8123-bcf76bdad253-kube-api-access-wbsdl\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.609209 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.609112 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rww\" (UniqueName: \"kubernetes.io/projected/2a85e903-5cfa-4bb0-b902-d510337216c9-kube-api-access-x8rww\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.609381 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.609358 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4pn\" (UniqueName: \"kubernetes.io/projected/1f2dabbc-40dc-4d65-b445-bcde680092ad-kube-api-access-jd4pn\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.609450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.609360 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zj4q\" (UniqueName: \"kubernetes.io/projected/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-kube-api-access-4zj4q\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.609509 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.609473 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzml\" (UniqueName: \"kubernetes.io/projected/9a4c7d5d-9276-4dce-8e0a-d7782b067792-kube-api-access-8vzml\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:17.609647 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.609628 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbxn\" (UniqueName: \"kubernetes.io/projected/1d5f7eda-1c08-4147-9fc8-241e0c982451-kube-api-access-cwbxn\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.714256 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.714226 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4lc2l" Nov 02 20:14:17.758256 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.758226 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:14:17.778897 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.778876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:17.783838 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.783817 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:14:17.792767 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.792735 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:14:17.810761 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.810733 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:17.821571 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.821550 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:14:17.831234 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.831216 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:14:17.836053 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.836037 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6zxch" Nov 02 20:14:17.884899 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:17.884874 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2dabbc_40dc_4d65_b445_bcde680092ad.slice/crio-7284ff950b032e64d654b0acfc8eb152fd125b412977f0e54cab683340f403c0 WatchSource:0}: Error finding container 7284ff950b032e64d654b0acfc8eb152fd125b412977f0e54cab683340f403c0: Status 404 returned error can't find the container with id 7284ff950b032e64d654b0acfc8eb152fd125b412977f0e54cab683340f403c0 Nov 02 20:14:17.885308 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:17.885295 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506cfda16cd111388fffa21903437d2c.slice/crio-0d17c413475fb29445320726e612e9b052e31784c1281e3732ff0e0c5d36c59a WatchSource:0}: Error finding container 0d17c413475fb29445320726e612e9b052e31784c1281e3732ff0e0c5d36c59a: Status 404 returned error can't find the container with id 0d17c413475fb29445320726e612e9b052e31784c1281e3732ff0e0c5d36c59a Nov 02 20:14:17.890801 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:17.890780 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:14:18.006922 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.006891 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51646e0acf14788341d5bd3c94d4017.slice/crio-eead1d1a71ef79ffcb5a6e63d34cbf3da2e3261ae3505ae71eca55daaf485a62 WatchSource:0}: Error finding container eead1d1a71ef79ffcb5a6e63d34cbf3da2e3261ae3505ae71eca55daaf485a62: Status 404 returned error can't find the container with id eead1d1a71ef79ffcb5a6e63d34cbf3da2e3261ae3505ae71eca55daaf485a62 Nov 02 20:14:18.008362 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.008321 2569 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:18.053045 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.053000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:18.053179 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.053086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:18.053179 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:18.053137 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:18.053179 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:18.053168 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:18.053293 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:18.053203 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret podName:6b02c000-6d43-4adf-ba8b-4afd3e1186bc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:19.053185451 +0000 UTC m=+3.341962382 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret") pod "global-pull-secret-syncer-s49d6" (UID: "6b02c000-6d43-4adf-ba8b-4afd3e1186bc") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:18.053293 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:18.053222 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs podName:9a4c7d5d-9276-4dce-8e0a-d7782b067792 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:19.053213431 +0000 UTC m=+3.341990353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs") pod "network-metrics-daemon-ndz76" (UID: "9a4c7d5d-9276-4dce-8e0a-d7782b067792") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:18.153997 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.153922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:18.154509 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:18.154049 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:14:18.154509 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:18.154065 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:14:18.154509 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:18.154074 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8s82w for pod openshift-network-diagnostics/network-check-target-89w6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:18.154509 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:18.154135 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w podName:75ef20c6-f775-4fc4-96e3-8bd5f4a983f2 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:19.154109629 +0000 UTC m=+3.442886566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8s82w" (UniqueName: "kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w") pod "network-check-target-89w6h" (UID: "75ef20c6-f775-4fc4-96e3-8bd5f4a983f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:18.191401 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.191375 2569 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:18.268750 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.268697 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"d51646e0acf14788341d5bd3c94d4017","Type":"ContainerStarted","Data":"eead1d1a71ef79ffcb5a6e63d34cbf3da2e3261ae3505ae71eca55daaf485a62"} Nov 02 20:14:18.269796 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.269770 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6zxch" event={"ID":"1f2dabbc-40dc-4d65-b445-bcde680092ad","Type":"ContainerStarted","Data":"7284ff950b032e64d654b0acfc8eb152fd125b412977f0e54cab683340f403c0"} Nov 02 20:14:18.270774 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.270754 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"506cfda16cd111388fffa21903437d2c","Type":"ContainerStarted","Data":"0d17c413475fb29445320726e612e9b052e31784c1281e3732ff0e0c5d36c59a"} Nov 02 20:14:18.383265 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.383242 2569 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:14:18.494724 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.494631 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 20:09:17 +0000 UTC" deadline="2028-08-13 06:09:58.087886562 +0000 UTC" Nov 02 20:14:18.494724 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:18.494656 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15585h55m39.59323353s" Nov 02 20:14:18.597106 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.597075 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c97247_cf8b_43b8_8fe4_4979ff070278.slice/crio-166a115e5de2c1dbccbc5db7da0885d9bd0be7de55b06a775bc120b8b1ac8418 WatchSource:0}: Error finding container 166a115e5de2c1dbccbc5db7da0885d9bd0be7de55b06a775bc120b8b1ac8418: Status 404 returned error can't find the container with id 166a115e5de2c1dbccbc5db7da0885d9bd0be7de55b06a775bc120b8b1ac8418 Nov 02 20:14:18.672036 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.672004 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b9f368_1f2b_41b8_a67a_6d17f4b9344c.slice/crio-9f2f56472fe4a6cd80dc8ec764bd26f5676f984d4abac9195e606f7b77ae1d3e WatchSource:0}: Error finding container 9f2f56472fe4a6cd80dc8ec764bd26f5676f984d4abac9195e606f7b77ae1d3e: Status 404 returned error can't find the container with id 9f2f56472fe4a6cd80dc8ec764bd26f5676f984d4abac9195e606f7b77ae1d3e Nov 02 20:14:18.682107 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.682086 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e28c5b_ebf2_41aa_9be9_52e5e31aafa6.slice/crio-92e499a486b158b6007853855bb2503b83f073a15b37fd0382625097bb827819 WatchSource:0}: Error finding container 92e499a486b158b6007853855bb2503b83f073a15b37fd0382625097bb827819: Status 404 returned error can't find the container with id 92e499a486b158b6007853855bb2503b83f073a15b37fd0382625097bb827819 Nov 02 20:14:18.693975 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.693947 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168a3d5f_16fa_44c2_8123_bcf76bdad253.slice/crio-5d963d561ca7b1047ba8401a18917f6e0df0bfe5b71baeb431d276e75670f17c WatchSource:0}: Error finding container 5d963d561ca7b1047ba8401a18917f6e0df0bfe5b71baeb431d276e75670f17c: Status 404 returned error can't find the container with id 5d963d561ca7b1047ba8401a18917f6e0df0bfe5b71baeb431d276e75670f17c Nov 02 20:14:18.701529 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.701507 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5f7eda_1c08_4147_9fc8_241e0c982451.slice/crio-b0a539aa1f85b275b7fefef1d17b8caced37a43a2600b776cf9179203b10a1cd WatchSource:0}: Error finding container b0a539aa1f85b275b7fefef1d17b8caced37a43a2600b776cf9179203b10a1cd: Status 404 returned error can't find the container with id b0a539aa1f85b275b7fefef1d17b8caced37a43a2600b776cf9179203b10a1cd Nov 02 20:14:18.711519 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.711494 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab012ae_9e57_45bb_8f50_9fe5f511a30c.slice/crio-c0aa140bb2df7507f3ab8de4b1678840fd8dbb248192bcfe134ab4cbd0cf4d8e WatchSource:0}: Error finding container c0aa140bb2df7507f3ab8de4b1678840fd8dbb248192bcfe134ab4cbd0cf4d8e: Status 404 returned error can't find the container with id c0aa140bb2df7507f3ab8de4b1678840fd8dbb248192bcfe134ab4cbd0cf4d8e Nov 02 20:14:18.712376 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.712353 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae28258_51a5_4e33_8040_39d8592006b8.slice/crio-dac0ff13d8ca56bec757c0fa7d01d7255ef736d2659fed84992e36c3f534da02 WatchSource:0}: Error finding container dac0ff13d8ca56bec757c0fa7d01d7255ef736d2659fed84992e36c3f534da02: Status 404 returned error can't find the container with id dac0ff13d8ca56bec757c0fa7d01d7255ef736d2659fed84992e36c3f534da02 Nov 02 20:14:18.732037 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:14:18.732017 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a85e903_5cfa_4bb0_b902_d510337216c9.slice/crio-208ff40c4ba58bbbd1c303d9837451ea37567b80ef280dbf88d53c175709b2c9 WatchSource:0}: Error finding container 208ff40c4ba58bbbd1c303d9837451ea37567b80ef280dbf88d53c175709b2c9: Status 404 returned error can't find the container with id 208ff40c4ba58bbbd1c303d9837451ea37567b80ef280dbf88d53c175709b2c9 Nov 02 20:14:19.059859 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.059817 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:19.060061 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.059974 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:19.060061 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.060012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:19.060061 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.060040 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret podName:6b02c000-6d43-4adf-ba8b-4afd3e1186bc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.060021666 +0000 UTC m=+5.348798614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret") pod "global-pull-secret-syncer-s49d6" (UID: "6b02c000-6d43-4adf-ba8b-4afd3e1186bc") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:19.060209 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.060128 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:19.060209 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.060178 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs podName:9a4c7d5d-9276-4dce-8e0a-d7782b067792 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.060162342 +0000 UTC m=+5.348939270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs") pod "network-metrics-daemon-ndz76" (UID: "9a4c7d5d-9276-4dce-8e0a-d7782b067792") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:19.161264 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.160974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:19.161264 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.161145 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:14:19.161264 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.161161 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:14:19.161264 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.161170 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8s82w for pod openshift-network-diagnostics/network-check-target-89w6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:19.161264 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.161223 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w podName:75ef20c6-f775-4fc4-96e3-8bd5f4a983f2 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:21.16120691 +0000 UTC m=+5.449983852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8s82w" (UniqueName: "kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w") pod "network-check-target-89w6h" (UID: "75ef20c6-f775-4fc4-96e3-8bd5f4a983f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:19.264944 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.264494 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:19.264944 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.264610 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:19.264944 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.264718 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:19.264944 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.264789 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:19.264944 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.264843 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:19.264944 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:19.264904 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:19.274106 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.274064 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnw5s" event={"ID":"0ae28258-51a5-4e33-8040-39d8592006b8","Type":"ContainerStarted","Data":"dac0ff13d8ca56bec757c0fa7d01d7255ef736d2659fed84992e36c3f534da02"} Nov 02 20:14:19.276186 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.276154 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-898kr" event={"ID":"eab012ae-9e57-45bb-8f50-9fe5f511a30c","Type":"ContainerStarted","Data":"c0aa140bb2df7507f3ab8de4b1678840fd8dbb248192bcfe134ab4cbd0cf4d8e"} Nov 02 20:14:19.277771 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.277715 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2l8lk" event={"ID":"168a3d5f-16fa-44c2-8123-bcf76bdad253","Type":"ContainerStarted","Data":"5d963d561ca7b1047ba8401a18917f6e0df0bfe5b71baeb431d276e75670f17c"} Nov 02 20:14:19.279117 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.279067 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4lc2l" event={"ID":"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6","Type":"ContainerStarted","Data":"92e499a486b158b6007853855bb2503b83f073a15b37fd0382625097bb827819"} Nov 02 20:14:19.281054 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.281027 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"208ff40c4ba58bbbd1c303d9837451ea37567b80ef280dbf88d53c175709b2c9"} Nov 02 20:14:19.282634 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.282608 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" event={"ID":"1d5f7eda-1c08-4147-9fc8-241e0c982451","Type":"ContainerStarted","Data":"b0a539aa1f85b275b7fefef1d17b8caced37a43a2600b776cf9179203b10a1cd"} Nov 02 20:14:19.283926 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.283902 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hm6m" event={"ID":"30b9f368-1f2b-41b8-a67a-6d17f4b9344c","Type":"ContainerStarted","Data":"9f2f56472fe4a6cd80dc8ec764bd26f5676f984d4abac9195e606f7b77ae1d3e"} Nov 02 20:14:19.285946 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.285917 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ks29" event={"ID":"31c97247-cf8b-43b8-8fe4-4979ff070278","Type":"ContainerStarted","Data":"166a115e5de2c1dbccbc5db7da0885d9bd0be7de55b06a775bc120b8b1ac8418"} Nov 02 20:14:19.494913 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.494786 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 20:09:17 +0000 UTC" deadline="2028-04-25 15:12:23.799740378 +0000 UTC" Nov 02 20:14:19.494913 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:19.494837 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12954h58m4.30490711s" Nov 02 20:14:21.079621 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:21.079586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:21.080248 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:21.079659 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:21.080248 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.079819 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:21.080248 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.079879 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs podName:9a4c7d5d-9276-4dce-8e0a-d7782b067792 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:25.079859151 +0000 UTC m=+9.368636092 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs") pod "network-metrics-daemon-ndz76" (UID: "9a4c7d5d-9276-4dce-8e0a-d7782b067792") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:21.080248 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.079946 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:21.080248 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.079979 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret podName:6b02c000-6d43-4adf-ba8b-4afd3e1186bc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:25.079967566 +0000 UTC m=+9.368744504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret") pod "global-pull-secret-syncer-s49d6" (UID: "6b02c000-6d43-4adf-ba8b-4afd3e1186bc") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:21.181106 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:21.181068 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:21.181273 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.181225 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:14:21.181273 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.181247 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:14:21.181379 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.181293 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8s82w for pod openshift-network-diagnostics/network-check-target-89w6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:21.181379 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.181355 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w podName:75ef20c6-f775-4fc4-96e3-8bd5f4a983f2 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:25.181336114 +0000 UTC m=+9.470113061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8s82w" (UniqueName: "kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w") pod "network-check-target-89w6h" (UID: "75ef20c6-f775-4fc4-96e3-8bd5f4a983f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:21.265226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:21.264672 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:21.265226 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.264831 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:21.265226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:21.264902 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:21.265226 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.264989 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:21.265226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:21.265036 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:21.265226 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:21.265109 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:23.265527 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:23.265293 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:23.265957 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:23.265657 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:23.265957 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:23.265772 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:23.265957 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:23.265856 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:23.265957 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:23.265927 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:23.266234 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:23.266014 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:23.311934 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:23.311898 2569 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="6e67e70f5ca11f0b29e18e3957f46748bd82c08fd1906b4527cad1e2c3b1f95b" exitCode=0 Nov 02 20:14:23.312105 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:23.311992 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ks29" event={"ID":"31c97247-cf8b-43b8-8fe4-4979ff070278","Type":"ContainerDied","Data":"6e67e70f5ca11f0b29e18e3957f46748bd82c08fd1906b4527cad1e2c3b1f95b"} Nov 02 20:14:23.314643 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:23.314612 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"d51646e0acf14788341d5bd3c94d4017","Type":"ContainerStarted","Data":"4271b54f71378a3ab00f8163c7889b42bb6e3f7b40aac9bce800c743cdc47d4c"} Nov 02 20:14:23.318102 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:23.318075 2569 generic.go:356] "Generic (PLEG): container finished" podID="506cfda16cd111388fffa21903437d2c" containerID="a2854149307d33177ecbe4a5857512779a487da6cbce62e0f567d52f42f95659" exitCode=0 Nov 02 20:14:23.318216 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:23.318117 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"506cfda16cd111388fffa21903437d2c","Type":"ContainerDied","Data":"a2854149307d33177ecbe4a5857512779a487da6cbce62e0f567d52f42f95659"} Nov 02 20:14:24.324430 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:24.324391 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnw5s" event={"ID":"0ae28258-51a5-4e33-8040-39d8592006b8","Type":"ContainerStarted","Data":"6db7a0c434772c0c2bc207f303c09abd5357ca7f422d6829c6a0543666744fc4"} Nov 02 20:14:24.432713 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:24.432007 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" podStartSLOduration=7.431987431 podStartE2EDuration="7.431987431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:23.656399252 +0000 UTC m=+7.945176197" watchObservedRunningTime="2026-11-02 20:14:24.431987431 +0000 UTC m=+8.720764374" Nov 02 20:14:25.119195 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.119153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:25.119369 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.119248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:25.119443 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.119399 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:25.119485 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.119465 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs podName:9a4c7d5d-9276-4dce-8e0a-d7782b067792 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.11944847 +0000 UTC m=+17.408225394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs") pod "network-metrics-daemon-ndz76" (UID: "9a4c7d5d-9276-4dce-8e0a-d7782b067792") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:25.119547 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.119524 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:25.119632 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.119553 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret podName:6b02c000-6d43-4adf-ba8b-4afd3e1186bc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.119543106 +0000 UTC m=+17.408320028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret") pod "global-pull-secret-syncer-s49d6" (UID: "6b02c000-6d43-4adf-ba8b-4afd3e1186bc") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:25.219864 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.219799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:25.220030 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.220015 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:14:25.220120 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.220037 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:14:25.220120 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.220049 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8s82w for pod openshift-network-diagnostics/network-check-target-89w6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:25.220120 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.220109 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w podName:75ef20c6-f775-4fc4-96e3-8bd5f4a983f2 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:33.220090365 +0000 UTC m=+17.508867305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8s82w" (UniqueName: "kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w") pod "network-check-target-89w6h" (UID: "75ef20c6-f775-4fc4-96e3-8bd5f4a983f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:25.265166 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.265127 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:25.265320 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.265273 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:25.265391 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.265372 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:25.265532 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.265478 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:25.265639 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.265622 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:25.265740 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:25.265720 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:25.330852 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.330819 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"506cfda16cd111388fffa21903437d2c","Type":"ContainerStarted","Data":"dc7a8404001189b5b5622f568d73481904e61899ce4055de6662f2968861eae9"} Nov 02 20:14:25.336104 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.335803 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8hm6m" event={"ID":"30b9f368-1f2b-41b8-a67a-6d17f4b9344c","Type":"ContainerStarted","Data":"73f7466ca01daaa36af1a5f2bb68a68bbd16af474dcf05387671bd6a3aaae9f5"} Nov 02 20:14:25.438530 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.438480 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wnw5s" podStartSLOduration=3.668662397 podStartE2EDuration="8.438465707s" totalImagesPullingTime="4.76980331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:24.539445001 +0000 UTC m=+8.828221943" watchObservedRunningTime="2026-11-02 20:14:25.438465707 +0000 UTC m=+9.727242644" Nov 02 20:14:25.439349 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:25.439298 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" podStartSLOduration=8.4390576 podStartE2EDuration="8.4390576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:25.438564193 +0000 UTC m=+9.727341136" watchObservedRunningTime="2026-11-02 20:14:25.4390576 +0000 UTC m=+9.727834543" Nov 02 20:14:26.156786 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:26.156704 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:26.157189 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:26.157168 2569 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:26.266756 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:26.265239 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8hm6m" podStartSLOduration=3.724533587 podStartE2EDuration="9.265218324s" totalImagesPullingTime="5.540684737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:25.548744564 +0000 UTC m=+9.837521509" watchObservedRunningTime="2026-11-02 20:14:26.265218324 +0000 UTC m=+10.553995267" Nov 02 20:14:26.338999 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:26.338921 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" event={"ID":"1d5f7eda-1c08-4147-9fc8-241e0c982451","Type":"ContainerStarted","Data":"321c373b7ad8b5f0d33eb4df3f5e2bc18a6fe807c87fecbfbc0b4e1bfea38c11"} Nov 02 20:14:26.345011 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:26.344979 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6zxch" event={"ID":"1f2dabbc-40dc-4d65-b445-bcde680092ad","Type":"ContainerStarted","Data":"8ed595e2f270356de18c07475080592ca9445a07ae3c3bc544225d30721009d7"} Nov 02 20:14:26.969123 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:26.969102 2569 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 20:14:27.210264 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.210110 2569 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T20:14:26.969118777Z","UUID":"27d2b472-c6b2-43aa-b416-b7a04e9cbc06","Handler":null,"Name":"","Endpoint":""} Nov 02 20:14:27.211817 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.211789 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 20:14:27.211817 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.211820 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 20:14:27.265425 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.265396 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:27.265571 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.265437 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:27.265571 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:27.265497 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:27.265692 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:27.265640 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:27.265745 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.265693 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:27.265796 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:27.265773 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:27.348723 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.348663 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" event={"ID":"1d5f7eda-1c08-4147-9fc8-241e0c982451","Type":"ContainerStarted","Data":"1e66f6f4d6a4bf10053d3d280646b6b382f716614490c7aa991406cce2559a67"} Nov 02 20:14:27.350535 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.350505 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4lc2l" event={"ID":"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6","Type":"ContainerStarted","Data":"9c14c264f2ed52e8ee06332c7eb52a627f1542ab1c32f0739a657e8c04827720"} Nov 02 20:14:27.350801 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.350779 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 20:14:27.460461 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.460366 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6zxch" podStartSLOduration=2.58388134 podStartE2EDuration="10.460352886s" totalImagesPullingTime="7.876471546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:26.452826687 +0000 UTC m=+10.741603631" watchObservedRunningTime="2026-11-02 20:14:27.460352886 +0000 UTC m=+11.749129873" Nov 02 20:14:27.460461 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:27.460445 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4lc2l" podStartSLOduration=3.509421802 podStartE2EDuration="11.460440874s" totalImagesPullingTime="7.951019072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:27.459892802 +0000 UTC m=+11.748669746" watchObservedRunningTime="2026-11-02 20:14:27.460440874 +0000 UTC m=+11.749217817" Nov 02 20:14:28.356472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:28.355630 2569 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="fd9eb2c17817046edf92586052f5f6dba7cf3dfc588a6a76cdf3732e44ffc44e" exitCode=0 Nov 02 20:14:28.356472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:28.355763 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ks29" event={"ID":"31c97247-cf8b-43b8-8fe4-4979ff070278","Type":"ContainerDied","Data":"fd9eb2c17817046edf92586052f5f6dba7cf3dfc588a6a76cdf3732e44ffc44e"} Nov 02 20:14:29.265138 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.265104 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:29.265273 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.265146 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:29.265273 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:29.265256 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:29.265346 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.265283 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:29.265376 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:29.265346 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:29.265406 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:29.265392 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:29.360062 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.360029 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" event={"ID":"1d5f7eda-1c08-4147-9fc8-241e0c982451","Type":"ContainerStarted","Data":"1e879959cebb4027f25e43c52b82819db9d678ae7cfa8ed028780cf7ade59539"} Nov 02 20:14:29.361764 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.361739 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-898kr" event={"ID":"eab012ae-9e57-45bb-8f50-9fe5f511a30c","Type":"ContainerStarted","Data":"696492ebd59bbc8566a1e7c41e285004469ef31507fcbc57e1d83a3d9db17e40"} Nov 02 20:14:29.365643 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.365611 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2l8lk" event={"ID":"168a3d5f-16fa-44c2-8123-bcf76bdad253","Type":"ContainerStarted","Data":"6bdae679c8c927e54273d5f925fa35dda0948785ead204e7f5a78082d399ecde"} Nov 02 20:14:29.470269 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.470214 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" podStartSLOduration=2.934920647 podStartE2EDuration="12.470195815s" totalImagesPullingTime="9.535275168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:29.469741498 +0000 UTC m=+13.758518440" watchObservedRunningTime="2026-11-02 20:14:29.470195815 +0000 UTC m=+13.758972759" Nov 02 20:14:29.578798 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.578710 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-898kr" podStartSLOduration=2.6991730350000003 podStartE2EDuration="12.578697441s" totalImagesPullingTime="9.879524406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:29.578519289 +0000 UTC m=+13.867296232" watchObservedRunningTime="2026-11-02 20:14:29.578697441 +0000 UTC m=+13.867474382" Nov 02 20:14:29.684930 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:29.684892 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2l8lk" podStartSLOduration=4.565444884 podStartE2EDuration="12.684878908s" totalImagesPullingTime="8.119434024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:29.684752313 +0000 UTC m=+13.973529257" watchObservedRunningTime="2026-11-02 20:14:29.684878908 +0000 UTC m=+13.973655851" Nov 02 20:14:30.368642 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:30.368607 2569 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="07e33c609da4e9e4930dced4365684b31dfd92364fee6681acb1f66d4f6494d0" exitCode=0 Nov 02 20:14:30.369111 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:30.368721 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ks29" event={"ID":"31c97247-cf8b-43b8-8fe4-4979ff070278","Type":"ContainerDied","Data":"07e33c609da4e9e4930dced4365684b31dfd92364fee6681acb1f66d4f6494d0"} Nov 02 20:14:31.265547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:31.265512 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:31.265547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:31.265534 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:31.265547 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:31.265554 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:31.265818 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:31.265640 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:31.265818 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:31.265765 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:31.265887 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:31.265834 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:32.036080 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:32.035818 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6zxch_1f2dabbc-40dc-4d65-b445-bcde680092ad/dns-node-resolver/0.log" Nov 02 20:14:32.375526 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:32.375441 2569 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="cb75ae3b26a8a627b594d05a001444c08ae17e40a0a1cfb8f96a0c0131f6dc02" exitCode=0 Nov 02 20:14:32.375526 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:32.375494 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ks29" event={"ID":"31c97247-cf8b-43b8-8fe4-4979ff070278","Type":"ContainerDied","Data":"cb75ae3b26a8a627b594d05a001444c08ae17e40a0a1cfb8f96a0c0131f6dc02"} Nov 02 20:14:32.892196 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:32.892148 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8hm6m_30b9f368-1f2b-41b8-a67a-6d17f4b9344c/node-ca/0.log" Nov 02 20:14:33.187294 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:33.187225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:33.187294 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:33.187271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:33.187699 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.187357 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:33.187699 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.187362 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:33.187699 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.187405 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret podName:6b02c000-6d43-4adf-ba8b-4afd3e1186bc nodeName:}" failed. No retries permitted until 2026-11-02 20:14:49.18739257 +0000 UTC m=+33.476169495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret") pod "global-pull-secret-syncer-s49d6" (UID: "6b02c000-6d43-4adf-ba8b-4afd3e1186bc") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:33.187699 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.187418 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs podName:9a4c7d5d-9276-4dce-8e0a-d7782b067792 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:49.18741149 +0000 UTC m=+33.476188411 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs") pod "network-metrics-daemon-ndz76" (UID: "9a4c7d5d-9276-4dce-8e0a-d7782b067792") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:33.265330 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:33.265247 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:33.265330 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:33.265275 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:33.265510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:33.265385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:33.265510 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.265393 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:33.265510 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.265462 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:33.265720 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.265535 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:33.287974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:33.287952 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:33.288107 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.288084 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:14:33.288107 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.288103 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:14:33.288196 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.288115 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8s82w for pod openshift-network-diagnostics/network-check-target-89w6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:33.288196 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:33.288165 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w podName:75ef20c6-f775-4fc4-96e3-8bd5f4a983f2 nodeName:}" failed. No retries permitted until 2026-11-02 20:14:49.288149954 +0000 UTC m=+33.576926880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8s82w" (UniqueName: "kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w") pod "network-check-target-89w6h" (UID: "75ef20c6-f775-4fc4-96e3-8bd5f4a983f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:35.265304 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:35.265229 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:35.265304 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:35.265272 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:35.265749 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:35.265330 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:35.265749 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:35.265347 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:35.265749 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:35.265398 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:35.265749 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:35.265464 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:35.382955 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:35.382924 2569 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="7a6097dafd9e22842d6cd1be6c19373be59e0fe5855bddb3a6b810a41f14d24b" exitCode=0 Nov 02 20:14:35.383090 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:35.382966 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ks29" event={"ID":"31c97247-cf8b-43b8-8fe4-4979ff070278","Type":"ContainerDied","Data":"7a6097dafd9e22842d6cd1be6c19373be59e0fe5855bddb3a6b810a41f14d24b"} Nov 02 20:14:36.387385 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:36.387345 2569 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="9716d8a6f31b50398651a89eec0dc6f6df15396258922f200368e2c6c39c9e0d" exitCode=0 Nov 02 20:14:36.387385 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:36.387389 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ks29" event={"ID":"31c97247-cf8b-43b8-8fe4-4979ff070278","Type":"ContainerDied","Data":"9716d8a6f31b50398651a89eec0dc6f6df15396258922f200368e2c6c39c9e0d"} Nov 02 20:14:37.264511 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:37.264476 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:37.264707 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:37.264553 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:37.264707 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:37.264643 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:37.264707 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:37.264652 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:37.264863 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:37.264740 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:37.264863 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:37.264820 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:37.393657 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:37.393619 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ks29" event={"ID":"31c97247-cf8b-43b8-8fe4-4979ff070278","Type":"ContainerStarted","Data":"852d3a9a806566a94d5424ab751bb8103aa3d24c16d4bb146366fd590e770170"} Nov 02 20:14:37.509374 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:37.509314 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6ks29" podStartSLOduration=8.781216459 podStartE2EDuration="21.509296348s" totalImagesPullingTime="12.728079889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:37.508994777 +0000 UTC m=+21.797771719" watchObservedRunningTime="2026-11-02 20:14:37.509296348 +0000 UTC m=+21.798073293" Nov 02 20:14:39.264992 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:39.264952 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:39.265470 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:39.264998 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:39.265470 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:39.264952 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:39.265470 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:39.265094 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:39.265470 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:39.265182 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:39.265470 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:39.265264 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:40.402724 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:40.402512 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"84268ffa05d1a130825522d08131fea3ffcc0e35eae58a891b02c7d7d50f5a42"} Nov 02 20:14:40.402987 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:40.402757 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"1992db6ccaffc6f5d4caeb0d22147946b8f1efbe831cadf20f0e2ef93d1fcd1c"} Nov 02 20:14:40.402987 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:40.402810 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"57ef1ae9aa874e2ba486173fccba96ed9b97864ac289a8d281ebf60900e323fc"} Nov 02 20:14:40.402987 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:40.402860 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"0e4e09bc69a60ccda51ed9481dd99e333eb38be329a03519b9d711d57d83298b"} Nov 02 20:14:40.402987 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:40.402906 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"eba34914611549e259e547e5f6a3867582ce4ebfb0aadfc361114e0b06a9e7d1"} Nov 02 20:14:41.265424 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:41.265388 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:41.265424 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:41.265416 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:41.265630 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:41.265506 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:41.265630 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:41.265531 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:41.265630 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:41.265591 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:41.265751 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:41.265632 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:41.410134 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:41.410100 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"a65f24f38f565427c69b3c32668c4ddc9ceac4e33b22a6e96e780a72d969c7d0"} Nov 02 20:14:43.264965 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:43.264933 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:43.264965 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:43.264968 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:43.265459 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:43.265069 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:43.265459 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:43.265129 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:43.265459 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:43.265201 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:43.265459 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:43.265270 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:43.417134 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:43.417108 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"235cb94a3feef8a4eebc791eb67d8be9d88e59ab0c72d6ed17d9f37363f9775e"} Nov 02 20:14:45.264519 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:45.264479 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:45.264519 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:45.264505 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:45.265272 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:45.264487 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:45.265272 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:45.264587 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:45.265272 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:45.264659 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:45.265272 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:45.264744 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:45.424141 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:45.424110 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" event={"ID":"2a85e903-5cfa-4bb0-b902-d510337216c9","Type":"ContainerStarted","Data":"c263a6450286f0a62a73b1bf8fa9f96ec80076a5d7c608b6136dd91c80de1002"} Nov 02 20:14:45.424386 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:45.424370 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:45.438558 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:45.438533 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:45.683099 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:45.683051 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" podStartSLOduration=8.310819247 podStartE2EDuration="29.683040378s" totalImagesPullingTime="21.372221131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:14:45.659012253 +0000 UTC m=+29.947789195" watchObservedRunningTime="2026-11-02 20:14:45.683040378 +0000 UTC m=+29.971817320" Nov 02 20:14:46.426536 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:46.426510 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:46.426960 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:46.426587 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:46.441236 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:46.441215 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:14:47.156892 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:47.156858 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:47.157082 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:47.157005 2569 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 20:14:47.157404 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:47.157386 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:14:47.265486 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:47.265454 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:47.265659 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:47.265576 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:47.265659 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:47.265599 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:47.265659 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:47.265596 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:47.265855 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:47.265685 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:47.265855 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:47.265748 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:49.199312 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:49.199279 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:49.199772 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:49.199342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:49.199772 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.199426 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:49.199772 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.199443 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:49.199772 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.199484 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret podName:6b02c000-6d43-4adf-ba8b-4afd3e1186bc nodeName:}" failed. No retries permitted until 2026-11-02 20:15:21.199470596 +0000 UTC m=+65.488247535 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret") pod "global-pull-secret-syncer-s49d6" (UID: "6b02c000-6d43-4adf-ba8b-4afd3e1186bc") : object "kube-system"/"original-pull-secret" not registered Nov 02 20:14:49.199772 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.199498 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs podName:9a4c7d5d-9276-4dce-8e0a-d7782b067792 nodeName:}" failed. No retries permitted until 2026-11-02 20:15:21.199492487 +0000 UTC m=+65.488269408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs") pod "network-metrics-daemon-ndz76" (UID: "9a4c7d5d-9276-4dce-8e0a-d7782b067792") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 20:14:49.264896 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:49.264866 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:49.264896 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:49.264883 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:49.265085 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:49.264907 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:49.265085 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.264974 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:49.265085 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.265015 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:49.265085 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.265075 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:49.300057 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:49.300031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:49.300192 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.300177 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 20:14:49.300232 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.300196 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 20:14:49.300232 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.300205 2569 projected.go:196] Error preparing data for projected volume kube-api-access-8s82w for pod openshift-network-diagnostics/network-check-target-89w6h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:49.300295 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:49.300244 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w podName:75ef20c6-f775-4fc4-96e3-8bd5f4a983f2 nodeName:}" failed. No retries permitted until 2026-11-02 20:15:21.30023106 +0000 UTC m=+65.589007998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8s82w" (UniqueName: "kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w") pod "network-check-target-89w6h" (UID: "75ef20c6-f775-4fc4-96e3-8bd5f4a983f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 20:14:51.265353 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:51.265184 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:51.265773 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:51.265259 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:51.265773 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:51.265421 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:51.265773 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:51.265268 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:51.265773 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:51.265504 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:51.265773 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:51.265556 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:53.264829 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:53.264800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:53.264829 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:53.264822 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:53.265255 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:53.264800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:53.265255 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:53.264891 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:53.265255 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:53.264971 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:53.265255 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:53.265021 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:55.265375 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.265348 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:55.265375 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.265371 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:55.265838 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.265385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:55.265838 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:55.265466 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:55.265838 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:55.265583 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:55.265838 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:55.265634 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:55.282074 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.282046 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s49d6"] Nov 02 20:14:55.285443 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.285415 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89w6h"] Nov 02 20:14:55.285597 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.285475 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ndz76"] Nov 02 20:14:55.443910 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.443880 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:55.444066 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.443880 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:55.444066 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:55.443979 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:55.444066 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:55.443887 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:55.444066 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:55.444039 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:55.444197 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:55.444117 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:57.265151 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:57.265118 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:57.265582 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:57.265194 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:57.265582 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:57.265275 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:57.265582 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:57.265281 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:14:57.265582 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:57.265379 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:57.265582 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:57.265438 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:59.264985 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:59.264953 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:14:59.265453 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:59.265052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:14:59.265453 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:59.265059 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-89w6h" podUID="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" Nov 02 20:14:59.265453 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:14:59.265073 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:14:59.265453 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:59.265174 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ndz76" podUID="9a4c7d5d-9276-4dce-8e0a-d7782b067792" Nov 02 20:14:59.265453 ip-10-0-134-206 kubenswrapper[2569]: E1102 20:14:59.265205 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s49d6" podUID="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" Nov 02 20:15:00.492821 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.492754 2569 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeReady" Nov 02 20:15:00.493323 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.492874 2569 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 20:15:00.636686 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.636515 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gjf9l"] Nov 02 20:15:00.659063 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.659035 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7rth8"] Nov 02 20:15:00.659386 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.659145 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.668859 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.668841 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x5lv8"] Nov 02 20:15:00.668957 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.668944 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:00.672911 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.672891 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gjf9l"] Nov 02 20:15:00.673026 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.672931 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rth8"] Nov 02 20:15:00.673026 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.673012 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.708129 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.708108 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 20:15:00.708129 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.708128 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-82tkg\"" Nov 02 20:15:00.708294 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.708149 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 20:15:00.719142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.719121 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 20:15:00.719978 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.719959 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x5lv8"] Nov 02 20:15:00.784472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784413 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c968f13c-daba-49a5-99a7-38063f221675-crio-socket\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.784472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784439 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c968f13c-daba-49a5-99a7-38063f221675-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.784472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784457 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvq5c\" (UniqueName: \"kubernetes.io/projected/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-kube-api-access-hvq5c\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:00.784655 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-tmp-dir\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.784655 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784520 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c968f13c-daba-49a5-99a7-38063f221675-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.784655 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-metrics-tls\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.784655 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784638 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knw45\" (UniqueName: \"kubernetes.io/projected/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-kube-api-access-knw45\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.784655 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784655 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vh67\" (UniqueName: \"kubernetes.io/projected/c968f13c-daba-49a5-99a7-38063f221675-kube-api-access-4vh67\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.784827 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784686 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-config-volume\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.784827 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c968f13c-daba-49a5-99a7-38063f221675-data-volume\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.784827 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.784779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-cert\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:00.809222 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.809179 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 20:15:00.809222 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.809199 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d6nbh\"" Nov 02 20:15:00.809222 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.809180 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 20:15:00.809364 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.809223 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 20:15:00.809364 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.809229 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 20:15:00.809364 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.809248 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 20:15:00.809364 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.809270 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 20:15:00.809364 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.809354 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-74k7h\"" Nov 02 20:15:00.885292 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-tmp-dir\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.885292 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c968f13c-daba-49a5-99a7-38063f221675-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.885450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885336 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-metrics-tls\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.885450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knw45\" (UniqueName: \"kubernetes.io/projected/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-kube-api-access-knw45\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.885450 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885368 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vh67\" (UniqueName: \"kubernetes.io/projected/c968f13c-daba-49a5-99a7-38063f221675-kube-api-access-4vh67\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.885596 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-config-volume\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.885596 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885542 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c968f13c-daba-49a5-99a7-38063f221675-data-volume\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.885596 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-cert\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:00.885763 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885635 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c968f13c-daba-49a5-99a7-38063f221675-crio-socket\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.885763 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885659 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c968f13c-daba-49a5-99a7-38063f221675-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.885763 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885704 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvq5c\" (UniqueName: \"kubernetes.io/projected/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-kube-api-access-hvq5c\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:00.885981 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885961 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c968f13c-daba-49a5-99a7-38063f221675-crio-socket\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.885981 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.885971 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-tmp-dir\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.886118 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.886006 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c968f13c-daba-49a5-99a7-38063f221675-data-volume\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.886355 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.886327 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-config-volume\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.886429 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.886387 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c968f13c-daba-49a5-99a7-38063f221675-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.889304 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.889281 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-metrics-tls\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.889393 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.889306 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c968f13c-daba-49a5-99a7-38063f221675-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.889393 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.889341 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-cert\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:00.942211 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.942182 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knw45\" (UniqueName: \"kubernetes.io/projected/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-kube-api-access-knw45\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.942638 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.942615 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vh67\" (UniqueName: \"kubernetes.io/projected/c968f13c-daba-49a5-99a7-38063f221675-kube-api-access-4vh67\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:00.942745 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.942731 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvq5c\" (UniqueName: \"kubernetes.io/projected/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-kube-api-access-hvq5c\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:00.978328 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.978309 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:00.982425 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.982409 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:00.994464 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:00.994436 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:01.265411 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.265380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:15:01.265411 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.265400 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:01.265616 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.265524 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:15:01.314601 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.314575 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4wc8w\"" Nov 02 20:15:01.314760 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.314589 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7bq62\"" Nov 02 20:15:01.314760 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.314717 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 20:15:01.314760 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.314730 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s29pz\"" Nov 02 20:15:01.314917 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.314765 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 20:15:01.314917 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.314787 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 20:15:01.314917 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.314823 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 20:15:01.404062 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.404036 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7rth8"] Nov 02 20:15:01.409945 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:01.409921 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aaa4ca9_71ad_4f88_898f_c6bc2799f20a.slice/crio-562612ccbfb1e40ce600a722a08dc3761a7d9eefce1f5b82438d5ec432e6ca4b WatchSource:0}: Error finding container 562612ccbfb1e40ce600a722a08dc3761a7d9eefce1f5b82438d5ec432e6ca4b: Status 404 returned error can't find the container with id 562612ccbfb1e40ce600a722a08dc3761a7d9eefce1f5b82438d5ec432e6ca4b Nov 02 20:15:01.432082 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.432054 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gjf9l"] Nov 02 20:15:01.435034 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.434944 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x5lv8"] Nov 02 20:15:01.435769 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:01.435745 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd8078a_c324_4fc8_b3d2_b631c6a2a724.slice/crio-dceba17a3f1e34b731836ba6fb50d5aa2fa4f84c93849f7c00fe48bd06d2a9d9 WatchSource:0}: Error finding container dceba17a3f1e34b731836ba6fb50d5aa2fa4f84c93849f7c00fe48bd06d2a9d9: Status 404 returned error can't find the container with id dceba17a3f1e34b731836ba6fb50d5aa2fa4f84c93849f7c00fe48bd06d2a9d9 Nov 02 20:15:01.437924 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:01.437900 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc968f13c_daba_49a5_99a7_38063f221675.slice/crio-2da82773ebeff9c10c719f6a6e3d1bcd6fe4ba5a635f8c0bb3e323af6ed3c218 WatchSource:0}: Error finding container 2da82773ebeff9c10c719f6a6e3d1bcd6fe4ba5a635f8c0bb3e323af6ed3c218: Status 404 returned error can't find the container with id 2da82773ebeff9c10c719f6a6e3d1bcd6fe4ba5a635f8c0bb3e323af6ed3c218 Nov 02 20:15:01.457619 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.457596 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rth8" event={"ID":"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a","Type":"ContainerStarted","Data":"562612ccbfb1e40ce600a722a08dc3761a7d9eefce1f5b82438d5ec432e6ca4b"} Nov 02 20:15:01.458575 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.458556 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gjf9l" event={"ID":"0cd8078a-c324-4fc8-b3d2-b631c6a2a724","Type":"ContainerStarted","Data":"dceba17a3f1e34b731836ba6fb50d5aa2fa4f84c93849f7c00fe48bd06d2a9d9"} Nov 02 20:15:01.459447 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:01.459428 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x5lv8" event={"ID":"c968f13c-daba-49a5-99a7-38063f221675","Type":"ContainerStarted","Data":"2da82773ebeff9c10c719f6a6e3d1bcd6fe4ba5a635f8c0bb3e323af6ed3c218"} Nov 02 20:15:02.464078 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:02.463848 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x5lv8" event={"ID":"c968f13c-daba-49a5-99a7-38063f221675","Type":"ContainerStarted","Data":"6bb437f15a96aaaa2c9ebb7da13fd22610084f628a1c9a58c01f1c9e3fbf31dd"} Nov 02 20:15:03.468851 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:03.468810 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x5lv8" event={"ID":"c968f13c-daba-49a5-99a7-38063f221675","Type":"ContainerStarted","Data":"74f2807f6eb198f0a8b9bd4828c50e516ac1b30db0594b7452e409c8cb91cd74"} Nov 02 20:15:04.474044 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:04.474011 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gjf9l" event={"ID":"0cd8078a-c324-4fc8-b3d2-b631c6a2a724","Type":"ContainerStarted","Data":"9f65a49a6535732adf1b72fb7aa7651e259af85efc413b84a1ad0cd877c5c8b6"} Nov 02 20:15:04.474471 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:04.474072 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gjf9l" event={"ID":"0cd8078a-c324-4fc8-b3d2-b631c6a2a724","Type":"ContainerStarted","Data":"e62fbd01e33ecc0072b354840ea485e63038f08904a7f5cf6fae295735036169"} Nov 02 20:15:04.474471 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:04.474287 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:04.584324 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:04.584206 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-gjf9l" podStartSLOduration=1.738562484 podStartE2EDuration="4.584186723s" totalImagesPullingTime="2.845624239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:04.583458292 +0000 UTC m=+48.872235237" watchObservedRunningTime="2026-11-02 20:15:04.584186723 +0000 UTC m=+48.872963667" Nov 02 20:15:05.479822 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:05.479784 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x5lv8" event={"ID":"c968f13c-daba-49a5-99a7-38063f221675","Type":"ContainerStarted","Data":"9f8e8f03b4aade328436fe746def2fba14677949bb6322032cfb54b808839a3a"} Nov 02 20:15:05.481161 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:05.481136 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7rth8" event={"ID":"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a","Type":"ContainerStarted","Data":"a3383cd334b22c6eab0d06bf3c6ee61d43c894ed5f78001a861d510850a94bfa"} Nov 02 20:15:05.591141 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:05.591090 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x5lv8" podStartSLOduration=1.90397308 podStartE2EDuration="5.591076433s" totalImagesPullingTime="3.687103353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:05.589953359 +0000 UTC m=+49.878730302" watchObservedRunningTime="2026-11-02 20:15:05.591076433 +0000 UTC m=+49.879853388" Nov 02 20:15:05.697878 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:05.697834 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7rth8" podStartSLOduration=2.165942863 podStartE2EDuration="5.697821817s" totalImagesPullingTime="3.531878954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:05.69753878 +0000 UTC m=+49.986315724" watchObservedRunningTime="2026-11-02 20:15:05.697821817 +0000 UTC m=+49.986598759" Nov 02 20:15:12.148107 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.148072 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh"] Nov 02 20:15:12.150577 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.150563 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.199557 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.199533 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 20:15:12.245797 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.245769 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 20:15:12.246081 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.246055 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 20:15:12.246214 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.246157 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4tnzz\"" Nov 02 20:15:12.246214 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.246181 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 20:15:12.246318 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.246247 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 20:15:12.247099 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.247082 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh"] Nov 02 20:15:12.259610 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.259585 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thktd\" (UniqueName: \"kubernetes.io/projected/d804b66e-c2da-42b6-add0-c11761995171-kube-api-access-thktd\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.259730 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.259614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d804b66e-c2da-42b6-add0-c11761995171-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.259730 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.259657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.259834 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.259818 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.292934 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.292905 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn"] Nov 02 20:15:12.295341 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.295324 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6tlrn"] Nov 02 20:15:12.295474 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.295457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.297536 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.297518 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.321747 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.321725 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn"] Nov 02 20:15:12.360510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.360485 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d804b66e-c2da-42b6-add0-c11761995171-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.360642 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.360552 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.360642 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.360621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.360775 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.360647 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thktd\" (UniqueName: \"kubernetes.io/projected/d804b66e-c2da-42b6-add0-c11761995171-kube-api-access-thktd\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.361130 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.361109 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d804b66e-c2da-42b6-add0-c11761995171-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.363370 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.363347 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.363428 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.363391 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.417470 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.417415 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thktd\" (UniqueName: \"kubernetes.io/projected/d804b66e-c2da-42b6-add0-c11761995171-kube-api-access-thktd\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.417691 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.417648 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 20:15:12.417792 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.417716 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 20:15:12.417792 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.417717 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 20:15:12.417792 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.417765 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 20:15:12.417897 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.417827 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvgts\"" Nov 02 20:15:12.417948 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.417935 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2nsqw\"" Nov 02 20:15:12.417980 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.417948 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 20:15:12.418039 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.418028 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 20:15:12.461555 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-textfile\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.461555 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461560 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-root\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.461703 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461580 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.461703 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9962v\" (UniqueName: \"kubernetes.io/projected/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-kube-api-access-9962v\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.461767 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461737 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.461803 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461773 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.461803 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461793 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.461858 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461812 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.461858 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461827 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2d2q\" (UniqueName: \"kubernetes.io/projected/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-api-access-v2d2q\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.461858 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461841 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-wtmp\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.461974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-sys\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.461974 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461929 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.462025 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.461972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-metrics-client-ca\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.462063 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.462049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.462107 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.462075 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-tls\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.462934 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.462922 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:12.562922 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.562893 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-textfile\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563058 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.562933 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-root\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563058 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.562993 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-root\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563058 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563023 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.563058 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9962v\" (UniqueName: \"kubernetes.io/projected/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-kube-api-access-9962v\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563221 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563087 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.563221 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.563221 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563129 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563221 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.563221 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563183 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2d2q\" (UniqueName: \"kubernetes.io/projected/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-api-access-v2d2q\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.563221 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-wtmp\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563226 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-sys\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563242 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563244 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-textfile\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-metrics-client-ca\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-tls\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-wtmp\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563413 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-sys\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.563466 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.563435 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.564091 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.564065 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.564183 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.564155 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.564183 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.564172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-metrics-client-ca\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.564454 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.564428 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.565432 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.565411 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.565842 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.565822 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.566282 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.566261 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-tls\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.566332 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.566318 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.619583 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.619561 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9962v\" (UniqueName: \"kubernetes.io/projected/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-kube-api-access-9962v\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.619660 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.619619 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2d2q\" (UniqueName: \"kubernetes.io/projected/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-api-access-v2d2q\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.869332 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.869295 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh"] Nov 02 20:15:12.873154 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:12.873129 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd804b66e_c2da_42b6_add0_c11761995171.slice/crio-22bac4646e71cf5ee72bcf0d266008029d24a36f738cc29e69a7a5cf035f470c WatchSource:0}: Error finding container 22bac4646e71cf5ee72bcf0d266008029d24a36f738cc29e69a7a5cf035f470c: Status 404 returned error can't find the container with id 22bac4646e71cf5ee72bcf0d266008029d24a36f738cc29e69a7a5cf035f470c Nov 02 20:15:12.909096 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.909075 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:12.914222 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:12.914201 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:12.922737 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:12.922711 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0a953f_8e98_4fd0_aaeb_0a450e0c1ac3.slice/crio-55e89fe3e687f542a384a773b510f27b59607f254bdcb24bec9ee5c00898e0da WatchSource:0}: Error finding container 55e89fe3e687f542a384a773b510f27b59607f254bdcb24bec9ee5c00898e0da: Status 404 returned error can't find the container with id 55e89fe3e687f542a384a773b510f27b59607f254bdcb24bec9ee5c00898e0da Nov 02 20:15:13.322205 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:13.322177 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn"] Nov 02 20:15:13.325078 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:13.325052 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7c610f_2bc8_45db_b648_3d7b4d3cee2b.slice/crio-c064fdf0dbba04a334dcb60c4ae08ff8ced20dde7540b2503943437534e6bd4d WatchSource:0}: Error finding container c064fdf0dbba04a334dcb60c4ae08ff8ced20dde7540b2503943437534e6bd4d: Status 404 returned error can't find the container with id c064fdf0dbba04a334dcb60c4ae08ff8ced20dde7540b2503943437534e6bd4d Nov 02 20:15:13.504092 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:13.504056 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" event={"ID":"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b","Type":"ContainerStarted","Data":"c064fdf0dbba04a334dcb60c4ae08ff8ced20dde7540b2503943437534e6bd4d"} Nov 02 20:15:13.505725 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:13.505701 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" event={"ID":"d804b66e-c2da-42b6-add0-c11761995171","Type":"ContainerStarted","Data":"517e792dce02f30df5c499e3a6b571ca4a8c25f9ee0e700eba1732af6056dd91"} Nov 02 20:15:13.505847 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:13.505756 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" event={"ID":"d804b66e-c2da-42b6-add0-c11761995171","Type":"ContainerStarted","Data":"23de47027c3c99d79687d64eecfbe68ab9eeaf265e2c75ee12b42c45c73a8edf"} Nov 02 20:15:13.505847 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:13.505784 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" event={"ID":"d804b66e-c2da-42b6-add0-c11761995171","Type":"ContainerStarted","Data":"22bac4646e71cf5ee72bcf0d266008029d24a36f738cc29e69a7a5cf035f470c"} Nov 02 20:15:13.506717 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:13.506697 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tlrn" event={"ID":"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3","Type":"ContainerStarted","Data":"55e89fe3e687f542a384a773b510f27b59607f254bdcb24bec9ee5c00898e0da"} Nov 02 20:15:14.483231 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.483207 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:14.514377 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.514325 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" event={"ID":"d804b66e-c2da-42b6-add0-c11761995171","Type":"ContainerStarted","Data":"ed9a4a6e84da4c10e2ff6ac30495e0285a4ff7a7f45ab26ed1f7e1e0fee6cfbe"} Nov 02 20:15:14.705047 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.704926 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" podStartSLOduration=1.292929417 podStartE2EDuration="2.70490514s" totalImagesPullingTime="1.411975723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:14.703755283 +0000 UTC m=+58.992532227" watchObservedRunningTime="2026-11-02 20:15:14.70490514 +0000 UTC m=+58.993682096" Nov 02 20:15:14.771112 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.771074 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t"] Nov 02 20:15:14.773890 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.773871 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.824498 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.824450 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 20:15:14.824722 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.824527 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 20:15:14.824722 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.824600 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 20:15:14.824909 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.824739 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 20:15:14.824909 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.824795 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jvjxk\"" Nov 02 20:15:14.824909 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.824802 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 20:15:14.824909 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.824853 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5ep5s2cc5pb2s\"" Nov 02 20:15:14.834360 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.834342 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t"] Nov 02 20:15:14.884333 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884310 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.884455 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884359 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.884455 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884384 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-grpc-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.884532 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884457 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.884532 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhhxh\" (UniqueName: \"kubernetes.io/projected/74280e02-7f3b-4a07-8bac-914610c5d04d-kube-api-access-hhhxh\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.884532 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884494 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.884532 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884511 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74280e02-7f3b-4a07-8bac-914610c5d04d-metrics-client-ca\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.884532 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.884722 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.884601 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74280e02-7f3b-4a07-8bac-914610c5d04d-tmp\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985212 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985139 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74280e02-7f3b-4a07-8bac-914610c5d04d-tmp\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985356 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985356 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985305 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985356 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985335 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-grpc-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985504 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985504 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985390 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhxh\" (UniqueName: \"kubernetes.io/projected/74280e02-7f3b-4a07-8bac-914610c5d04d-kube-api-access-hhhxh\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985504 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985504 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985443 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74280e02-7f3b-4a07-8bac-914610c5d04d-metrics-client-ca\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985504 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985470 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.985751 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.985593 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74280e02-7f3b-4a07-8bac-914610c5d04d-tmp\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.986637 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.986588 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74280e02-7f3b-4a07-8bac-914610c5d04d-metrics-client-ca\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.988221 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.988195 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.988347 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.988195 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.988409 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.988392 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.988801 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.988773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-grpc-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.989036 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.989013 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:14.989126 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:14.989106 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:15.042994 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.042968 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhxh\" (UniqueName: \"kubernetes.io/projected/74280e02-7f3b-4a07-8bac-914610c5d04d-kube-api-access-hhhxh\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:15.092981 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.092953 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:15.519538 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.519503 2569 generic.go:356] "Generic (PLEG): container finished" podID="aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3" containerID="ba462a0249ea84347053453e9410fccd4ee8ddf7f254a50968e54bb591b862a8" exitCode=0 Nov 02 20:15:15.522573 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.522473 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tlrn" event={"ID":"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3","Type":"ContainerDied","Data":"ba462a0249ea84347053453e9410fccd4ee8ddf7f254a50968e54bb591b862a8"} Nov 02 20:15:15.523048 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.523017 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" event={"ID":"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b","Type":"ContainerStarted","Data":"1af0a1692263c8cf483bb65815b92d863bc501ab494ec14d20e020c1434e3340"} Nov 02 20:15:15.523161 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.523083 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" event={"ID":"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b","Type":"ContainerStarted","Data":"ea44c08dd038fdbef4b0d880cdfa9a89babbea75b7b0058238f0ac05a21478df"} Nov 02 20:15:15.523161 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.523111 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" event={"ID":"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b","Type":"ContainerStarted","Data":"27f875fb327cc99c605f237117ab64eb142f99a250f1b30a238f04522a7c429c"} Nov 02 20:15:15.555402 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.555379 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t"] Nov 02 20:15:15.604951 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:15.604926 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74280e02_7f3b_4a07_8bac_914610c5d04d.slice/crio-e1b4b83378b045a9b2143a38a8d596aec4a6e8a35bf04e653ee63b5c225f9a2f WatchSource:0}: Error finding container e1b4b83378b045a9b2143a38a8d596aec4a6e8a35bf04e653ee63b5c225f9a2f: Status 404 returned error can't find the container with id e1b4b83378b045a9b2143a38a8d596aec4a6e8a35bf04e653ee63b5c225f9a2f Nov 02 20:15:15.743018 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:15.742963 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" podStartSLOduration=1.791098788 podStartE2EDuration="3.742930689s" totalImagesPullingTime="1.951831901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:15.742391524 +0000 UTC m=+60.031168465" watchObservedRunningTime="2026-11-02 20:15:15.742930689 +0000 UTC m=+60.031707628" Nov 02 20:15:16.527242 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:16.527200 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" event={"ID":"74280e02-7f3b-4a07-8bac-914610c5d04d","Type":"ContainerStarted","Data":"e1b4b83378b045a9b2143a38a8d596aec4a6e8a35bf04e653ee63b5c225f9a2f"} Nov 02 20:15:16.529586 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:16.529554 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tlrn" event={"ID":"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3","Type":"ContainerStarted","Data":"1d6a6eb1e0d287d9c90f3d3dc7e8f693b321c2417fa22904da74132a9d08cc11"} Nov 02 20:15:16.529728 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:16.529604 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tlrn" event={"ID":"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3","Type":"ContainerStarted","Data":"0813305a05f21a5e4f21ab0774010e46772b3779bd86fb38107465f954aa9873"} Nov 02 20:15:16.641978 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:16.641763 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6tlrn" podStartSLOduration=3.03516333 podStartE2EDuration="4.641747042s" totalImagesPullingTime="1.532982547s" totalInitContainerRuntime="73.601165ms" isStatefulPod=true podCreationTimestamp="2026-11-02 20:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:16.64056597 +0000 UTC m=+60.929342913" watchObservedRunningTime="2026-11-02 20:15:16.641747042 +0000 UTC m=+60.930523984" Nov 02 20:15:16.963690 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:16.963641 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p"] Nov 02 20:15:16.966103 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:16.966079 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:17.014879 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.014848 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k9j84\"" Nov 02 20:15:17.015056 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.014892 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 20:15:17.022357 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.022333 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p"] Nov 02 20:15:17.102968 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.102932 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3824767d-0c5a-420e-9541-0ca551ee1e24-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:17.103122 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.103056 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4ht\" (UniqueName: \"kubernetes.io/projected/3824767d-0c5a-420e-9541-0ca551ee1e24-kube-api-access-nf4ht\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:17.204142 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.204105 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3824767d-0c5a-420e-9541-0ca551ee1e24-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:17.204317 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.204209 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4ht\" (UniqueName: \"kubernetes.io/projected/3824767d-0c5a-420e-9541-0ca551ee1e24-kube-api-access-nf4ht\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:17.206438 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.206415 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3824767d-0c5a-420e-9541-0ca551ee1e24-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:17.260984 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.260922 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4ht\" (UniqueName: \"kubernetes.io/projected/3824767d-0c5a-420e-9541-0ca551ee1e24-kube-api-access-nf4ht\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:17.277798 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.277774 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:17.643490 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:17.643455 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p"] Nov 02 20:15:17.647379 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:17.647339 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3824767d_0c5a_420e_9541_0ca551ee1e24.slice/crio-660bf9f5a55737482b05797181392636e59ca8d95332bd226ed263cee90fecd0 WatchSource:0}: Error finding container 660bf9f5a55737482b05797181392636e59ca8d95332bd226ed263cee90fecd0: Status 404 returned error can't find the container with id 660bf9f5a55737482b05797181392636e59ca8d95332bd226ed263cee90fecd0 Nov 02 20:15:18.440598 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:18.440574 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:18.541120 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:18.541091 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" event={"ID":"74280e02-7f3b-4a07-8bac-914610c5d04d","Type":"ContainerStarted","Data":"bda5d5826dcebbbde103611c93629be9a67adf0ea96382bf94a4b9f1f9b06059"} Nov 02 20:15:18.541264 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:18.541167 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" event={"ID":"74280e02-7f3b-4a07-8bac-914610c5d04d","Type":"ContainerStarted","Data":"e9e2885cd6541a39ee348e5c0eb34513240b3fb16ec0ea71bcd3846e7c272a38"} Nov 02 20:15:18.541264 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:18.541212 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" event={"ID":"74280e02-7f3b-4a07-8bac-914610c5d04d","Type":"ContainerStarted","Data":"fd3e491e8257f0411970bc57848a6bc4adf7228eca1c8cf2c2f3b8de679936e9"} Nov 02 20:15:18.542227 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:18.542211 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" event={"ID":"3824767d-0c5a-420e-9541-0ca551ee1e24","Type":"ContainerStarted","Data":"660bf9f5a55737482b05797181392636e59ca8d95332bd226ed263cee90fecd0"} Nov 02 20:15:19.547373 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:19.547339 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" event={"ID":"74280e02-7f3b-4a07-8bac-914610c5d04d","Type":"ContainerStarted","Data":"8fadf18e2043b2afdce31ff2f1d52f99280accfc5d0dd4b7b6a37aca09910326"} Nov 02 20:15:19.548808 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:19.548781 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" event={"ID":"3824767d-0c5a-420e-9541-0ca551ee1e24","Type":"ContainerStarted","Data":"24f40f22a9e440ef9fad46a34f2d130275801ef2b201961ab0574ee4e6830730"} Nov 02 20:15:19.548968 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:19.548951 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:19.553545 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:19.553527 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:19.656839 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:19.656791 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" podStartSLOduration=1.875302477 podStartE2EDuration="3.656776352s" totalImagesPullingTime="1.781473875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:19.655925477 +0000 UTC m=+63.944702422" watchObservedRunningTime="2026-11-02 20:15:19.656776352 +0000 UTC m=+63.945553295" Nov 02 20:15:20.554534 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:20.554495 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" event={"ID":"74280e02-7f3b-4a07-8bac-914610c5d04d","Type":"ContainerStarted","Data":"a9ccc15be7e465f61dfd805334647ead26f8dca29ef71e20cb03abb4f3a6c9c7"} Nov 02 20:15:20.554964 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:20.554587 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" event={"ID":"74280e02-7f3b-4a07-8bac-914610c5d04d","Type":"ContainerStarted","Data":"6507b61bc29cb7cc2c33b60fb18b0916a0d16b1b2c798434b41aa8021520ba20"} Nov 02 20:15:20.670808 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:20.670754 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" podStartSLOduration=2.978187792 podStartE2EDuration="6.670740465s" totalImagesPullingTime="3.692552673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:15:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:20.669412178 +0000 UTC m=+64.958189121" watchObservedRunningTime="2026-11-02 20:15:20.670740465 +0000 UTC m=+64.959517407" Nov 02 20:15:21.130585 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.130550 2569 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:15:21.133412 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.133387 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.182384 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.182360 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 20:15:21.182507 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.182399 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 20:15:21.182559 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.182513 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aa2ub3sv54fn3\"" Nov 02 20:15:21.228018 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228000 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 20:15:21.228104 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228090 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dfwqd\"" Nov 02 20:15:21.228165 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228130 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 20:15:21.228165 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228137 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 20:15:21.228256 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228165 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 20:15:21.228256 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228243 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 20:15:21.228347 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228260 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 20:15:21.228347 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228291 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 20:15:21.228347 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.228324 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 20:15:21.233933 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.233914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn5dr\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234004 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.233942 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234004 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.233960 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234004 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.233980 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234088 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234045 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234088 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234146 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234146 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234203 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234185 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:21.234235 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234203 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234235 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234217 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234235 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234232 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234330 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234293 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234330 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234321 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234414 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234445 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234426 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234487 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234475 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234518 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234495 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234545 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234522 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234574 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234574 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.234629 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.234601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:15:21.320899 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.320871 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 20:15:21.321200 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.321175 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 20:15:21.321722 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.321700 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 20:15:21.321722 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.321699 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 20:15:21.321872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.321723 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 20:15:21.321872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.321758 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 20:15:21.321872 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.321783 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 20:15:21.322646 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.322632 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:15:21.326601 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.326576 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:21.326910 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.326895 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:15:21.334983 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.334962 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335057 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.334990 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335057 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335057 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335023 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335057 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335047 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335085 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335127 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:15:21.335226 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335204 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5dr\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335227 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335275 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335378 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335410 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335432 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.335472 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.335460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.336210 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.336108 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.336633 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.336605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.337185 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.337144 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.337744 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.337722 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.337956 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.337931 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.338543 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.338522 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.338740 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.338654 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.338888 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.338866 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.339388 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.339247 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.339388 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.339345 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.339524 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.339402 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.339524 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.339417 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.339714 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.339692 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.340209 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.340184 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.340765 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.340741 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.340969 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.340948 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.341078 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.341058 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.341817 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.341801 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.341988 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.341968 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.385445 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.385393 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 20:15:21.394698 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.394658 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5dr\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.434864 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.434845 2569 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 20:15:21.439950 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.439930 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:15:21.451930 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.451909 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:21.472798 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.472776 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s29pz\"" Nov 02 20:15:21.475300 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.475277 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:15:21.484425 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.484409 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7bq62\"" Nov 02 20:15:21.487467 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.487446 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:21.499654 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.499635 2569 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4wc8w\"" Nov 02 20:15:21.502798 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.502759 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:15:21.562964 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.562942 2569 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:21.870841 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.870688 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:15:21.873434 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:21.873405 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dbeb5e_76a3_4e56_9843_60aaeb743e39.slice/crio-6b4b83c9cc89dc8a01bf188266c0fe664964175b508c0adf9b9092e97ff001f0 WatchSource:0}: Error finding container 6b4b83c9cc89dc8a01bf188266c0fe664964175b508c0adf9b9092e97ff001f0: Status 404 returned error can't find the container with id 6b4b83c9cc89dc8a01bf188266c0fe664964175b508c0adf9b9092e97ff001f0 Nov 02 20:15:21.899600 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.899578 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ndz76"] Nov 02 20:15:21.902224 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:21.902202 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4c7d5d_9276_4dce_8e0a_d7782b067792.slice/crio-66814c4de9447e1571930fdb20c3867410771c46116e316142607a8a8bb0fe32 WatchSource:0}: Error finding container 66814c4de9447e1571930fdb20c3867410771c46116e316142607a8a8bb0fe32: Status 404 returned error can't find the container with id 66814c4de9447e1571930fdb20c3867410771c46116e316142607a8a8bb0fe32 Nov 02 20:15:21.906175 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.906152 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s49d6"] Nov 02 20:15:21.908976 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:21.908956 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b02c000_6d43_4adf_ba8b_4afd3e1186bc.slice/crio-7431250c2191f92e57f287ae3ebd91dc0c6566a9e73c8c9b6fe9eeb3d568d650 WatchSource:0}: Error finding container 7431250c2191f92e57f287ae3ebd91dc0c6566a9e73c8c9b6fe9eeb3d568d650: Status 404 returned error can't find the container with id 7431250c2191f92e57f287ae3ebd91dc0c6566a9e73c8c9b6fe9eeb3d568d650 Nov 02 20:15:21.909063 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:21.909013 2569 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-89w6h"] Nov 02 20:15:21.918823 ip-10-0-134-206 kubenswrapper[2569]: W1102 20:15:21.918799 2569 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ef20c6_f775_4fc4_96e3_8bd5f4a983f2.slice/crio-5ecccf676c96eef4a7e265553e6e99eb4058c21a4984efcf21d9f425b8077843 WatchSource:0}: Error finding container 5ecccf676c96eef4a7e265553e6e99eb4058c21a4984efcf21d9f425b8077843: Status 404 returned error can't find the container with id 5ecccf676c96eef4a7e265553e6e99eb4058c21a4984efcf21d9f425b8077843 Nov 02 20:15:22.566960 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:22.566925 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s49d6" event={"ID":"6b02c000-6d43-4adf-ba8b-4afd3e1186bc","Type":"ContainerStarted","Data":"7431250c2191f92e57f287ae3ebd91dc0c6566a9e73c8c9b6fe9eeb3d568d650"} Nov 02 20:15:22.568198 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:22.568175 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerStarted","Data":"6b4b83c9cc89dc8a01bf188266c0fe664964175b508c0adf9b9092e97ff001f0"} Nov 02 20:15:22.569264 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:22.569240 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89w6h" event={"ID":"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2","Type":"ContainerStarted","Data":"5ecccf676c96eef4a7e265553e6e99eb4058c21a4984efcf21d9f425b8077843"} Nov 02 20:15:22.570294 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:22.570278 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndz76" event={"ID":"9a4c7d5d-9276-4dce-8e0a-d7782b067792","Type":"ContainerStarted","Data":"66814c4de9447e1571930fdb20c3867410771c46116e316142607a8a8bb0fe32"} Nov 02 20:15:23.581221 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:23.579494 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerStarted","Data":"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda"} Nov 02 20:15:24.586935 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:24.586896 2569 generic.go:356] "Generic (PLEG): container finished" podID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" exitCode=0 Nov 02 20:15:24.587385 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:24.587000 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerDied","Data":"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda"} Nov 02 20:15:25.595028 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:25.594989 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndz76" event={"ID":"9a4c7d5d-9276-4dce-8e0a-d7782b067792","Type":"ContainerStarted","Data":"8109ecc371730c86b23bc977011ac77cb4a5a6e80d8750eeff4923b09fb63017"} Nov 02 20:15:25.595400 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:25.595081 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ndz76" event={"ID":"9a4c7d5d-9276-4dce-8e0a-d7782b067792","Type":"ContainerStarted","Data":"0ab9a9310f753e80911d4a1792b9884de0839a251ba2505d61fdf5da9fb5481a"} Nov 02 20:15:25.704510 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:25.704440 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ndz76" podStartSLOduration=65.9183989 podStartE2EDuration="1m8.704420882s" totalImagesPullingTime="2.786021982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:25.702752525 +0000 UTC m=+69.991529471" watchObservedRunningTime="2026-11-02 20:15:25.704420882 +0000 UTC m=+69.993197823" Nov 02 20:15:26.601463 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:26.601416 2569 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-89w6h" event={"ID":"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2","Type":"ContainerStarted","Data":"62ea5943a9fe39df7308e9c05d8e659002d1dff278318ec6ba4cd28cb35881ec"} Nov 02 20:15:26.710593 ip-10-0-134-206 kubenswrapper[2569]: I1102 20:15:26.710538 2569 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-89w6h" podStartSLOduration=65.806499229 podStartE2EDuration="1m9.710521043s" totalImagesPullingTime="3.904021814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:15:26.708735219 +0000 UTC m=+70.997512162" watchObservedRunningTime="2026-11-02 20:15:26.710521043 +0000 UTC m=+70.999297986" Nov 02 20:15:27.562302 ip-10-0-134-206 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 20:15:27.570097 ip-10-0-134-206 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 20:15:27.570333 ip-10-0-134-206 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 20:15:27.570546 ip-10-0-134-206 systemd[1]: kubelet.service: Consumed 3.752s CPU time. Nov 02 20:15:27.585291 ip-10-0-134-206 systemd[1]: Starting Kubernetes Kubelet... Nov 02 20:15:27.673345 ip-10-0-134-206 kubenswrapper[5420]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:15:27.673345 ip-10-0-134-206 kubenswrapper[5420]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 20:15:27.673345 ip-10-0-134-206 kubenswrapper[5420]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:15:27.673345 ip-10-0-134-206 kubenswrapper[5420]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 20:15:27.675753 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675650 5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:15:27.675753 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675753 5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:15:27.675753 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675757 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:15:27.675753 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675761 5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675764 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675767 5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675770 5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675773 5420 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675777 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675780 5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675783 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675785 5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675788 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675790 5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675802 5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675805 5420 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675808 5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675812 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675816 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675821 5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675825 5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675829 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675831 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:15:27.675902 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675834 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675836 5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675839 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675842 5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675845 5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675848 5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675850 5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675852 5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675855 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675858 5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675861 5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675863 5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675866 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675869 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675872 5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675875 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675877 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675881 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675886 5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:15:27.676465 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675893 5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675898 5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675903 5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675907 5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675911 5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675915 5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675920 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675927 5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675932 5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675936 5420 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675941 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675945 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675948 5420 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675951 5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675953 5420 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675956 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675959 5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675961 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675964 5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675966 5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:15:27.677191 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675969 5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675972 5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675974 5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675977 5420 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675979 5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675982 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675985 5420 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675988 5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675991 5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675993 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675996 5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.675998 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676001 5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676004 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676008 5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676013 5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676017 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676021 5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676025 5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:15:27.677984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676029 5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676032 5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676040 5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676047 5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676052 5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676057 5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676062 5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676067 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676072 5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676078 5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676084 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676089 5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676093 5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676096 5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676098 5420 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676101 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676104 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676106 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676109 5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:15:27.678780 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676114 5420 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676117 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676119 5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676122 5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676125 5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676127 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.676997 5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677005 5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677008 5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677011 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677014 5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677017 5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677020 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677022 5420 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677025 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677028 5420 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677031 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677033 5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677036 5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:15:27.679546 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677039 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677041 5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677044 5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677046 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677050 5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677052 5420 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677055 5420 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677058 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677061 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677071 5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677076 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677080 5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677084 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677087 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677089 5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677092 5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677096 5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677100 5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677104 5420 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677106 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:15:27.680400 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677109 5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677111 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677113 5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677116 5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677119 5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677122 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677124 5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677127 5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677129 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677134 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677137 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677140 5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677142 5420 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677145 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677148 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677150 5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677153 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677156 5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677158 5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:15:27.681238 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677161 5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677163 5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677165 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677168 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677171 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677173 5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677176 5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677178 5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677181 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677183 5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677185 5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677188 5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677191 5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677194 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677197 5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677199 5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677202 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677206 5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677211 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677215 5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:15:27.682033 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677219 5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677223 5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677227 5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677231 5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677235 5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677239 5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677243 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677247 5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677252 5420 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677255 5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677259 5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677264 5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677268 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677271 5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677277 5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677284 5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677289 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677293 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677297 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:15:27.682873 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677302 5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677306 5420 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677310 5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677315 5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677319 5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677323 5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677327 5420 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677331 5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677336 5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677340 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677344 5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677348 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677352 5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677356 5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677360 5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677910 5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677919 5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677923 5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677928 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677932 5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:15:27.683657 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677936 5420 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677940 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677944 5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677948 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677952 5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677957 5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677960 5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677963 5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677966 5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677968 5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677971 5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677973 5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677976 5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677979 5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677981 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677984 5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677987 5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677992 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677995 5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.677997 5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:15:27.684479 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678000 5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678003 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678005 5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678008 5420 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678015 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678019 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678024 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678027 5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678031 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678036 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678040 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678044 5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678049 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678054 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678058 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678062 5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678067 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678069 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678072 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:15:27.685325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678074 5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678077 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678080 5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678082 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678085 5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678087 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678090 5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678092 5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678097 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678099 5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678103 5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678106 5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678109 5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678112 5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678116 5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678120 5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678124 5420 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678128 5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678139 5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:15:27.686046 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678143 5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678147 5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678150 5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678155 5420 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678159 5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678162 5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678164 5420 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678167 5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678169 5420 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678172 5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678174 5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678177 5420 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678181 5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678184 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678186 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678189 5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678191 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678194 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678196 5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:15:27.686534 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678200 5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678203 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678208 5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678211 5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678214 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678217 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678220 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678222 5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678225 5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678227 5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678230 5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678232 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678235 5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678241 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678244 5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678247 5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678249 5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678252 5420 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678254 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678257 5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:15:27.687219 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678260 5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678262 5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678265 5420 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.678267 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678395 5420 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678406 5420 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678414 5420 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678419 5420 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678423 5420 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678426 5420 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678430 5420 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678435 5420 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678438 5420 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678441 5420 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678448 5420 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678451 5420 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678455 5420 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678458 5420 flags.go:64] FLAG: --cgroup-root="" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678460 5420 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678464 5420 flags.go:64] FLAG: --client-ca-file="" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678466 5420 flags.go:64] FLAG: --cloud-provider="external" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678469 5420 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678477 5420 flags.go:64] FLAG: --cluster-domain="" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678479 5420 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 20:15:27.687892 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678482 5420 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678485 5420 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678494 5420 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678498 5420 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678501 5420 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678504 5420 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678507 5420 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678512 5420 flags.go:64] FLAG: --contention-profiling="false" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678516 5420 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678521 5420 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678525 5420 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678530 5420 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678536 5420 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678540 5420 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678544 5420 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678549 5420 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678554 5420 flags.go:64] FLAG: --enable-server="true" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678557 5420 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678564 5420 flags.go:64] FLAG: --event-burst="100" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678567 5420 flags.go:64] FLAG: --event-qps="50" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678569 5420 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678572 5420 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678577 5420 flags.go:64] FLAG: --eviction-hard="" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678581 5420 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 20:15:27.688593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678584 5420 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678587 5420 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678590 5420 flags.go:64] FLAG: --eviction-soft="" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678593 5420 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678596 5420 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678598 5420 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678601 5420 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678604 5420 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678607 5420 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678609 5420 flags.go:64] FLAG: --feature-gates="" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678614 5420 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678622 5420 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678625 5420 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678628 5420 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678631 5420 flags.go:64] FLAG: --healthz-port="10248" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678635 5420 flags.go:64] FLAG: --help="false" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678637 5420 flags.go:64] FLAG: --hostname-override="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678640 5420 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678644 5420 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678647 5420 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678650 5420 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678653 5420 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678656 5420 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 20:15:27.689373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678659 5420 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678661 5420 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678664 5420 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678680 5420 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678685 5420 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678689 5420 flags.go:64] FLAG: --kube-reserved="" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678694 5420 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678699 5420 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678705 5420 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678707 5420 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678710 5420 flags.go:64] FLAG: --lock-file="" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678713 5420 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678716 5420 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678719 5420 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678726 5420 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678730 5420 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678735 5420 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678739 5420 flags.go:64] FLAG: --logging-format="text" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678743 5420 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678748 5420 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678752 5420 flags.go:64] FLAG: --manifest-url="" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678759 5420 flags.go:64] FLAG: --manifest-url-header="" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678765 5420 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678770 5420 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678775 5420 flags.go:64] FLAG: --max-pods="110" Nov 02 20:15:27.690228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678778 5420 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678781 5420 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678784 5420 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678787 5420 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678790 5420 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678794 5420 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678797 5420 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678804 5420 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678808 5420 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678811 5420 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678814 5420 flags.go:64] FLAG: --pod-cidr="" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678816 5420 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678819 5420 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678822 5420 flags.go:64] FLAG: --pods-per-core="0" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678826 5420 flags.go:64] FLAG: --port="10250" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678830 5420 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678834 5420 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0c184f5e24f5baa55" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678837 5420 flags.go:64] FLAG: --qos-reserved="" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678840 5420 flags.go:64] FLAG: --read-only-port="10255" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678843 5420 flags.go:64] FLAG: --register-node="true" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678845 5420 flags.go:64] FLAG: --register-with-taints="" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678852 5420 flags.go:64] FLAG: --registry-burst="10" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678855 5420 flags.go:64] FLAG: --registry-qps="5" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678858 5420 flags.go:64] FLAG: --reserved-cpus="" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678861 5420 flags.go:64] FLAG: --reserved-memory="" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678865 5420 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 20:15:27.690880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678868 5420 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678871 5420 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678874 5420 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678877 5420 flags.go:64] FLAG: --runonce="false" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678880 5420 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678883 5420 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678886 5420 flags.go:64] FLAG: --seccomp-default="false" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678889 5420 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678892 5420 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678894 5420 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678897 5420 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678900 5420 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678903 5420 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678906 5420 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678909 5420 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678912 5420 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678917 5420 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678921 5420 flags.go:64] FLAG: --system-cgroups="" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678926 5420 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678933 5420 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678939 5420 flags.go:64] FLAG: --tls-cert-file="" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678944 5420 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678950 5420 flags.go:64] FLAG: --tls-min-version="" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678955 5420 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678959 5420 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 20:15:27.691643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678963 5420 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678966 5420 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678970 5420 flags.go:64] FLAG: --v="2" Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678977 5420 flags.go:64] FLAG: --version="false" Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678984 5420 flags.go:64] FLAG: --vmodule="" Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678989 5420 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.678994 5420 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679162 5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679171 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679177 5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679182 5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679188 5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679192 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679196 5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679199 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679201 5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679204 5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679207 5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679209 5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679212 5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679215 5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:15:27.692597 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679217 5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679220 5420 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679222 5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679225 5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679227 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679230 5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679234 5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679238 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679241 5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679243 5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679246 5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679248 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679250 5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679253 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679255 5420 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679259 5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679263 5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679267 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679272 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:15:27.693383 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679276 5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679281 5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679285 5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679289 5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679293 5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679297 5420 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679301 5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679306 5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679310 5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679313 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679317 5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679322 5420 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679327 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679332 5420 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679336 5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679341 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679346 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679350 5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679355 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:15:27.693984 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679362 5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679368 5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679371 5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679373 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679376 5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679379 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679381 5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679384 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679387 5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679390 5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679393 5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679395 5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679398 5420 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679400 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679403 5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679405 5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679407 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679410 5420 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679413 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679416 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:15:27.694592 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679419 5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679421 5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679424 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679426 5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679429 5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679431 5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679433 5420 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679436 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679438 5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679441 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679443 5420 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679446 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679449 5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679454 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679456 5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679459 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679461 5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679465 5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679469 5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679473 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:15:27.695307 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679477 5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679481 5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679485 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679489 5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679493 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679497 5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679501 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679505 5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679509 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679512 5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679515 5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679520 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679524 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679528 5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.679567 5420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679603 5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:15:27.696062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679607 5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679611 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679614 5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679616 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679619 5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679622 5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679624 5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679627 5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679629 5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679636 5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679640 5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679645 5420 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679649 5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679653 5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679656 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679660 5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679664 5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679680 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679685 5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:15:27.696568 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679689 5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679693 5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679696 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679698 5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679701 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679703 5420 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679708 5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679713 5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679718 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679725 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679730 5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679734 5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679739 5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679744 5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679749 5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679755 5420 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679759 5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679763 5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679767 5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:15:27.697222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679769 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679773 5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679777 5420 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679783 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679786 5420 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679789 5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679792 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679795 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679797 5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679800 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679802 5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679805 5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679807 5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679810 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679812 5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679815 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679818 5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679820 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679823 5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:15:27.697958 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679825 5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679828 5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679831 5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679833 5420 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679837 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679840 5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679844 5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679849 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679852 5420 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679856 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679860 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679864 5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679868 5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679872 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679876 5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679880 5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679887 5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679892 5420 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679896 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679900 5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:15:27.698547 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679904 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679909 5420 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679913 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679918 5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679923 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679927 5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679932 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679934 5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679937 5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679940 5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679942 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679945 5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679948 5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679951 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679954 5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679956 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679958 5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679962 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679964 5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679967 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:15:27.699156 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679970 5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679972 5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679975 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679977 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679980 5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679983 5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679986 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.679988 5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.684391 5420 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.684411 5420 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.684434 5420 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.684441 5420 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684496 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684501 5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684504 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684507 5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684510 5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684513 5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684516 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:15:27.699813 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684518 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684521 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684524 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684526 5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684529 5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684532 5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684534 5420 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684537 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684540 5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684543 5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684545 5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684548 5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684551 5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684553 5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684557 5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684561 5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684566 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684571 5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684576 5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684579 5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:15:27.700417 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684581 5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684584 5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684587 5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684589 5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684592 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684595 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684598 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684601 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684604 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684608 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684612 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684615 5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684618 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684621 5420 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684624 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684626 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684629 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684632 5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684634 5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:15:27.701055 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684636 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684639 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684641 5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684644 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684646 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684649 5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684651 5420 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684654 5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684656 5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684658 5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684661 5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684684 5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684689 5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684693 5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684698 5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684700 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684703 5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684705 5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684708 5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684712 5420 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:15:27.701600 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684714 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684717 5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684719 5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684722 5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684727 5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684731 5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684735 5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684740 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684744 5420 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684747 5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684750 5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684752 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684755 5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684758 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684762 5420 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684766 5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684771 5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684774 5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684777 5420 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684781 5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:15:27.702317 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684784 5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684787 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684790 5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684793 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684797 5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684800 5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684803 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684806 5420 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684809 5420 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684811 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684814 5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684816 5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684819 5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684823 5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684826 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684829 5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684833 5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684836 5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684840 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:15:27.702953 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.684845 5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.684873 5420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685043 5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685048 5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685052 5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685054 5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685057 5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685060 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685062 5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685065 5420 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685075 5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685078 5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685080 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685082 5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685085 5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685088 5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:15:27.703564 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685090 5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685093 5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685135 5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.685173 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686415 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686426 5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686431 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686436 5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686441 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686445 5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686449 5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686452 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686455 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686458 5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686461 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686464 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686467 5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686473 5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686475 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686478 5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:15:27.704079 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686481 5420 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686483 5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686486 5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686489 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686493 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686495 5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686498 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686500 5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686503 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686508 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686511 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686514 5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686516 5420 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686519 5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686522 5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686524 5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686527 5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686531 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686533 5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:15:27.704713 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686536 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686545 5420 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686550 5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686558 5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686562 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686569 5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686576 5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686582 5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686586 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686590 5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686595 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686600 5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686604 5420 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686609 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686614 5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686623 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686626 5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686629 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686634 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686637 5420 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:15:27.705345 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686640 5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686643 5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686645 5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686648 5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686651 5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686653 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686656 5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686659 5420 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686664 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686680 5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686685 5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686713 5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686718 5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686722 5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686725 5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686728 5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686732 5420 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686734 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686737 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:15:27.705991 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686740 5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686746 5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686748 5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686751 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686754 5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686756 5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686759 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686761 5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686764 5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686767 5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686769 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686774 5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686777 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686779 5420 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.686818 5420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686919 5420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 20:15:27.706556 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686925 5420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686929 5420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686932 5420 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686936 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686938 5420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686941 5420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686944 5420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686950 5420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686953 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686956 5420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686959 5420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686961 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686964 5420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686966 5420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686969 5420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686972 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686974 5420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686977 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686979 5420 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686984 5420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 20:15:27.707045 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686987 5420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686990 5420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686992 5420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686995 5420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.686998 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687000 5420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687002 5420 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687008 5420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687013 5420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687017 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687022 5420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687027 5420 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687037 5420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687041 5420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687046 5420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687051 5420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687055 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687060 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687064 5420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687067 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 20:15:27.707650 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687070 5420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687073 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687076 5420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687079 5420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687082 5420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687087 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687089 5420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687092 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687095 5420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687097 5420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687100 5420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687102 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687105 5420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687107 5420 feature_gate.go:386] unrecognized feature gate: Example Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687110 5420 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687113 5420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687115 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687122 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687125 5420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687129 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 20:15:27.708262 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687132 5420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687134 5420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687137 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687140 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687142 5420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687145 5420 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687147 5420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687150 5420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687153 5420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687156 5420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687161 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687163 5420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687166 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687170 5420 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687174 5420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687180 5420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687189 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687217 5420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687222 5420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 20:15:27.708887 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687226 5420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687229 5420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687232 5420 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687238 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687240 5420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687243 5420 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687246 5420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687249 5420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687251 5420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687254 5420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687256 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687261 5420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687264 5420 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687267 5420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687270 5420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687275 5420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687277 5420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687280 5420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687283 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687285 5420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 20:15:27.709540 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687288 5420 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687290 5420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687293 5420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687295 5420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687298 5420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:15:27.687304 5420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.687385 5420 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.688837 5420 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.688890 5420 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.689176 5420 server.go:1014] "Starting client certificate rotation" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.689336 5420 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.689386 5420 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 20:09:16 +0000 UTC" deadline="2028-06-12 01:33:20.256815691 +0000 UTC" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.689427 5420 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14093h17m52.567391403s" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.690157 5420 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.691459 5420 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.692839 5420 log.go:25] "Validated CRI v1 runtime API" Nov 02 20:15:27.710500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.694972 5420 log.go:25] "Validated CRI v1 image API" Nov 02 20:15:27.711188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.695771 5420 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 20:15:27.711188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.698204 5420 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9f7e624d-14aa-49af-97ae-565ab1507b7a:/dev/nvme0n1p3 dfce8e62-9211-47ba-b230-100974542e48:/dev/nvme0n1p4] Nov 02 20:15:27.711311 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.698229 5420 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d17c413475fb29445320726e612e9b052e31784c1281e3732ff0e0c5d36c59a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d17c413475fb29445320726e612e9b052e31784c1281e3732ff0e0c5d36c59a/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/166a115e5de2c1dbccbc5db7da0885d9bd0be7de55b06a775bc120b8b1ac8418/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/166a115e5de2c1dbccbc5db7da0885d9bd0be7de55b06a775bc120b8b1ac8418/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/208ff40c4ba58bbbd1c303d9837451ea37567b80ef280dbf88d53c175709b2c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/208ff40c4ba58bbbd1c303d9837451ea37567b80ef280dbf88d53c175709b2c9/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22bac4646e71cf5ee72bcf0d266008029d24a36f738cc29e69a7a5cf035f470c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22bac4646e71cf5ee72bcf0d266008029d24a36f738cc29e69a7a5cf035f470c/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2da82773ebeff9c10c719f6a6e3d1bcd6fe4ba5a635f8c0bb3e323af6ed3c218/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2da82773ebeff9c10c719f6a6e3d1bcd6fe4ba5a635f8c0bb3e323af6ed3c218/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55e89fe3e687f542a384a773b510f27b59607f254bdcb24bec9ee5c00898e0da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55e89fe3e687f542a384a773b510f27b59607f254bdcb24bec9ee5c00898e0da/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/562612ccbfb1e40ce600a722a08dc3761a7d9eefce1f5b82438d5ec432e6ca4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/562612ccbfb1e40ce600a722a08dc3761a7d9eefce1f5b82438d5ec432e6ca4b/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d963d561ca7b1047ba8401a18917f6e0df0bfe5b71baeb431d276e75670f17c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d963d561ca7b1047ba8401a18917f6e0df0bfe5b71baeb431d276e75670f17c/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ecccf676c96eef4a7e265553e6e99eb4058c21a4984efcf21d9f425b8077843/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ecccf676c96eef4a7e265553e6e99eb4058c21a4984efcf21d9f425b8077843/userdata/shm major:0 minor:388 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/660bf9f5a55737482b05797181392636e59ca8d95332bd226ed263cee90fecd0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/660bf9f5a55737482b05797181392636e59ca8d95332bd226ed263cee90fecd0/userdata/shm major:0 minor:326 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66814c4de9447e1571930fdb20c3867410771c46116e316142607a8a8bb0fe32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66814c4de9447e1571930fdb20c3867410771c46116e316142607a8a8bb0fe32/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b4b83c9cc89dc8a01bf188266c0fe664964175b508c0adf9b9092e97ff001f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b4b83c9cc89dc8a01bf188266c0fe664964175b508c0adf9b9092e97ff001f0/userdata/shm major:0 minor:382 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7284ff950b032e64d654b0acfc8eb152fd125b412977f0e54cab683340f403c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7284ff950b032e64d654b0acfc8eb152fd125b412977f0e54cab683340f403c0/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7431250c2191f92e57f287ae3ebd91dc0c6566a9e73c8c9b6fe9eeb3d568d650/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7431250c2191f92e57f287ae3ebd91dc0c6566a9e73c8c9b6fe9eeb3d568d650/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92e499a486b158b6007853855bb2503b83f073a15b37fd0382625097bb827819/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92e499a486b158b6007853855bb2503b83f073a15b37fd0382625097bb827819/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f2f56472fe4a6cd80dc8ec764bd26f5676f984d4abac9195e606f7b77ae1d3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f2f56472fe4a6cd80dc8ec764bd26f5676f984d4abac9195e606f7b77ae1d3e/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0a539aa1f85b275b7fefef1d17b8caced37a43a2600b776cf9179203b10a1cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0a539aa1f85b275b7fefef1d17b8caced37a43a2600b776cf9179203b10a1cd/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c064fdf0dbba04a334dcb60c4ae08ff8ced20dde7540b2503943437534e6bd4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c064fdf0dbba04a334dcb60c4ae08ff8ced20dde7540b2503943437534e6bd4d/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dac0ff13d8ca56bec757c0fa7d01d7255ef736d2659fed84992e36c3f534da02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dac0ff13d8ca56bec757c0fa7d01d7255ef736d2659fed84992e36c3f534da02/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dceba17a3f1e34b731836ba6fb50d5aa2fa4f84c93849f7c00fe48bd06d2a9d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dceba17a3f1e34b731836ba6fb50d5aa2fa4f84c93849f7c00fe48bd06d2a9d9/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1b4b83378b045a9b2143a38a8d596aec4a6e8a35bf04e653ee63b5c225f9a2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1b4b83378b045a9b2143a38a8d596aec4a6e8a35bf04e653ee63b5c225f9a2f/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eead1d1a71ef79ffcb5a6e63d34cbf3da2e3261ae3505ae71eca55daaf485a62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eead1d1a71ef79ffcb5a6e63d34cbf3da2e3261ae3505ae71eca55daaf485a62/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ae28258-51a5-4e33-8040-39d8592006b8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0ae28258-51a5-4e33-8040-39d8592006b8/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cd8078a-c324-4fc8-b3d2-b631c6a2a724/volumes/kubernetes.io~projected/kube-api-access-knw45:{mountpoint:/var/lib/kubelet/pods/0cd8078a-c324-4fc8-b3d2-b631c6a2a724/volumes/kubernetes.io~projected/kube-api-access-knw45 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cd8078a-c324-4fc8-b3d2-b631c6a2a724/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0cd8078a-c324-4fc8-b3d2-b631c6a2a724/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/168a3d5f-16fa-44c2-8123-bcf76bdad253/volumes/kubernetes.io~projected/kube-api-access-wbsdl:{mountpoint:/var/lib/kubelet/pods/168a3d5f-16fa-44c2-8123-bcf76bdad253/volumes/kubernetes.io~projected/kube-api-access-wbsdl major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d5f7eda-1c08-4147-9fc8-241e0c982451/volumes/kubernetes.io~projected/kube-api-access-cwbxn:{mountpoint:/var/lib/kubelet/pods/1d5f7eda-1c08-4147-9fc8-241e0c982451/volumes/kubernetes.io~projected/kube-api-access-cwbxn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f2dabbc-40dc-4d65-b445-bcde680092ad/volumes/kubernetes.io~projected/kube-api-access-jd4pn:{mountpoint:/var/lib/kubelet/pods/1f2dabbc-40dc-4d65-b445-bcde680092ad/volumes/kubernetes.io~projected/kube-api-access-jd4pn major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~empty-dir/config-out major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~projected/kube-api-access-jn5dr:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~projected/kube-api-access-jn5dr major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~projected/tls-assets major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/config major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/web-config major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volumes/kubernetes.io~projected/kube-api-access-x8rww:{mountpoint:/var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volumes/kubernetes.io~projected/kube-api-access-x8rww major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30b9f368-1f2b-41b8-a67a-6d17f4b9344c/volumes/kubernetes.io~projected/kube-api-access-4zj4q:{mountpoint:/var/lib/kubelet/pods/30b9f368-1f2b-41b8-a67a-6d17f4b9344c/volumes/kubernetes.io~projected/kube-api-access-4zj4q major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31c97247-cf8b-43b8-8fe4-4979ff070278/volumes/kubernetes.io~projected/kube-api-access-7cvrw:{mountpoint:/var/lib/kubelet/pods/31c97247-cf8b-43b8-8fe4-4979ff070278/volumes/kubernetes.io~projected/kube-api-access-7cvrw major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6/volumes/kubernetes.io~projected/kube-api-access-n25wg:{mountpoint:/var/lib/kubelet/pods/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6/volumes/kubernetes.io~projected/kube-api-access-n25wg major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3824767d-0c5a-420e-9541-0ca551ee1e24/volumes/kubernetes.io~projected/kube-api-access-nf4ht:{mountpoint:/var/lib/kubelet/pods/3824767d-0c5a-420e-9541-0ca551ee1e24/volumes/kubernetes.io~projected/kube-api-access-nf4ht major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3824767d-0c5a-420e-9541-0ca551ee1e24/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/3824767d-0c5a-420e-9541-0ca551ee1e24/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b02c000-6d43-4adf-ba8b-4afd3e1186bc/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6b02c000-6d43-4adf-ba8b-4afd3e1186bc/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~projected/kube-api-access-hhhxh:{mountpoint:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~projected/kube-api-access-hhhxh major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2/volumes/kubernetes.io~projected/kube-api-access-8s82w:{mountpoint:/var/lib/kubelet/pods/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2/volumes/kubernetes.io~projected/kube-api-access-8s82w major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a/volumes/kubernetes.io~projected/kube-api-access-hvq5c:{mountpoint:/var/lib/kubelet/pods/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a/volumes/kubernetes.io~projected/kube-api-access-hvq5c major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a/volumes/kubernetes.io~secret/cert major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a4c7d5d-9276-4dce-8e0a-d7782b067792/volumes/kubernetes.io~projected/kube-api-access-8vzml:{mountpoint:/var/lib/kubelet/pods/9a4c7d5d-9276-4dce-8e0a-d7782b067792/volumes/kubernetes.io~projected/kube-api-access-8vzml major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a4c7d5d-9276-4dce-8e0a-d7782b067792/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9a4c7d5d-9276-4dce-8e0a-d7782b067792/volumes/kubernetes.io~secret/metrics-certs major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~projected/kube-api-access-9962v:{mountpoint:/var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~projected/kube-api-access-9962v major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~projected/kube-api-access-v2d2q:{mountpoint:/var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~projected/kube-api-access-v2d2q major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c968f13c-daba-49a5-99a7-38063f221675/volumes/kubernetes.io~projected/kube-api-access-4vh67:{mountpoint:/var/lib/kubelet/pods/c968f13c-daba-49a5-99a7-38063f221675/volumes/kubernetes.io~projected/kube-api-access-4vh67 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c968f13c-daba-49a5-99a7-38063f221675/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/c968f13c-daba-49a5-99a7-38063f221675/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~projected/kube-api-access-thktd:{mountpoint:/var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~projected/kube-api-access-thktd major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~projected/kube-api-access-49gqp:{mountpoint:/var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~projected/kube-api-access-49gqp major:0 minor:51 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/26c1b2862e51d4863039fce93d9f1427f777417d417724c57c18606b885bd208/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c098df097be828f9aa8ba64742779a1467279b09bade29b709e1f75e687d1dd8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6f2e7e802b05fe4c7c826b605d358a512f9e3bcaab0d8571329c87c4dce8553c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/6afaf4c82dca2993451ea0201644df26a24e8b6e6b2c4f4938822923c8d17966/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/018fdb8d9cc248f4e68c44f69099b14f0aed447f17ba23e3452c4e6dd6ee7d8a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/2fd585dce8742801e7c1b783435279fe56ba92f4538514d018edbe7fd509b66b/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7754dbef78693716b9e76152fe31cfd3c042d5c4aa9df5bcee94dd6322c182d0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e8e59f4cc4b384ed0f2ab75933edb66fb53057c2456ecc78b77c3fa868814c45/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/918b69138a6df44b807ab7d496d769da8367e4bb8bcb68a952a6f3534fb2d9c0/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c4a3dbd2c56bfb9af76e824d12adab56d77282b11ab272f9e64dc3cb294a88c5/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/f8991ff0b7c241055181e6ce1c32c61934abb89e9149d95e495718e19139c1bd/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/3c636bb969c94d5cca0ba12627591d0554ea3c52705bc3351663aaa3e94cf6f0/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/08817c20151ea97d6eb76f40d3f9022299ad480623b80c6eef22993d8eb8dd45/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/165485ba5016e1d885ec6fc614d61e8cea25230a44d7c53272cadb754f2dd111/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/05ca677d5c6cb162bb8f72b1edee5999de359585df14356721ac276a23d2dc13/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c635c5592a3616201b70a65dce3dd285a6c79e384d33a4a393759d7fe8bdb0ce/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/658403ab5cdab0d973f51104ea1a3420b0c8f49d6b1a96aa916a27d2b2d77467/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e9f75990b5cc190ba56bb4ee2fa072193a0a626035ca32b93a2c5d14820b6df8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7c314f7652f9f2dcaa43fdf7385f9f5fbec181ead44e06149f30eb70299fed22/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4918ae8e13382f80dbfb9aef49d9032cfbd950567db88f6eed4f6ae89ed34d7d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c0f167620cd4f87737d63a84a4497fec73b07be23fa15425d5f7b8e4e92ea78d/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b8dbbfdaa6fed936b0ee93a46b342d993f1cd381c7761d332832cb6f104830a1/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/b25518ea8bec8a9110cee2fb9c8718484f562845a04132edc43ac62733780758/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/5e21f5e95115e72fa0efc99203d52011acfe5ec3c4fd1ac9b45f2affc8653531/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/acdf8bcf98b8af6a7113b9e899679f524f0f787e0d299f83ee9fa260986364cd/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/0312adaa7676ba3e077bb0875f664062b2da05e358343e6513912615c61cd7b0/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/df7417a41021227cd28dc971d7995564eef4f8333bfae5a4a83172e69350889d/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4388a2d2504a43594ceaa12bb94acd1c29843add36758a15ab1189103c92f0ea/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/bd8f763b70bbacf5ed83f4a69769b8381ff7242998fc242647459a841bd77657/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/ab441fcb2c7339b363a862052986a666443004f83ae79701ad809c316225a766/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/50dca3135e08cb99e127d49453865cd75931330015c91ffed9d6fe51fca0f539/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/08458c7027198245dd2867597d651102c36241b95a6d8e11cdfdec32ded4d1b4/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/efe4456d97abda0f3880ac846015606f042797893668bdf9e9f9ee448442d406/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/f6e18114ad1231d3d09d096fd762a902072e583060753c41c8683f5f02b36a91/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/07d9e4ce33e80d2fd14dee6f0c543f5d04a590d25b099f4ae22f16237725e2db/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/6b2baec58d0b27b2220e63b503cfca7ff023e4d6e87195d51083b7e3721f061c/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/de493f4b4bb8be530e117e9026fe15fdd11e336bc30ee509b147b75ea82c9530/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/9c96a451d17e9846acb475b6e7a3509586e315a7f5cd6d4d854e430eff3ab64d/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/e5c2af072048c285e93ff5305a614ad03698a0638e20bae639c3ece408109ea4/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/eea982625972d772e60b3c68d317975699427124a010e20523dd5e8c7037f976/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/8a6cfb7794ee31455e7b6772625f6b5fa657fb0d89b06e1dd9052bd363a15616/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/5258ce6d559561d7e79a8455cc1830f192c4055091743fe69b4822a3fd4120be/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/0bfc0f2ed767c9affccd66353984c066970f7177a4a034d77b834d5ca401e93c/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/124e138804eedbd4a92c3153d67e5fd6c37d08cca3a2a7e5d683cdfbc85e402d/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/e96f0751fce3c9888ad4790caebe43036da5adc23bf7c0b1ec816c27fa143809/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/4a72a54467a959780d573b35ca50cd721487c9e0237f38b016410ebfc7626858/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/2f04f9e94a6a230957c08a38648a22d108290ef8ece3e2d75255bf297d546156/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/163cf72433e10dfb7b306202d70e321a8b5ee699e35dd96648c5a840bcb5a2de/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/40f0936ff1b862cb3f9471794d8c26217f67e838f2710b6cd9aea2f35859d879/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/75ba34a35ba11bd062587aa533c2f8e4425926667a0086d7210354b4b8ccb067/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/4ec29ee33e4f9c267726530d47891af2c7a663714bb591e0d4ebd2d524bc594e/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/c22ce2bb95b9920c988c937c32f6a0ed8ea9c1d708c496d4c207b6226f663668/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/afe3b7d2a7689946e80f5c7ad86838595f8e6b4f6ed0ecb7164757fc761c8741/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/82c57540da4fbb4080d62fe2d6bb0610b631e67adccb5294f525ca4210760af5/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/36a49f5accd3a86c26b457dd5cb86e6df73a081846afbe3f10c532238223bd1e/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/bedc469fb5d70840fec4a9cfcdd97181032b8b6df7cdfdfd6c48cf4ac241f6ba/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/a1fb694898f27359f345bc970cd8e96b075ee648e53636263bb465f0d74c8c7f/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/0746fb2b487d844877e14f543f1b0cb4858e785184874800eccbeb24b1699c19/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/661ccf8ced9fc4e0ac16a9f4e7400c2d2e5c2e5a2ab6ddcd6bc8e58c58623570/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/2f3a3194ddf77504b0b8336d838490194dcf8c594f222e8c39aca04fd5d17832/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/cbd59811e7aa670b02eae7f5ec1d1d5d5999363fd9a163433dc79dd2824a7f5a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/db3b7feeb60e6a7b3ef52c3377bb6fa1a1b109d52c8d38478269214e62f4350c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/324d45bc79001106a948b5abf56daebc715e6f1dde98d962236dc3c69de192d3/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/68e22d8c313ad4acde90be445810fb36cf7a3765b5128d74565d8396173f17ec/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/51c99adbc26873df3c08dac3834e8e46f0390d6e2aa1911b816a4b8b7269741b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ea0d44c036d39069898b2a46468f23ceae7f4141bc33574d60a0d9ae3399d62d/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/20e593d325e9706716fbb803ed2d8ebe9d367528ffa582231e87437bdd10ae0b/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a949e290ff792f54f3cfa05ce95190d7ac5050bef444b83c8a44972d47e3a139/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.707518 5420 manager.go:217] Machine: {Timestamp:2026-11-02 20:15:27.706899964 +0000 UTC m=+0.071511165 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199867 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e45c1b6f838d062e5f90aadd8200 SystemUUID:ec26e45c-1b6f-838d-062e-5f90aadd8200 BootID:7f137620-0796-4c4d-a993-9a37d09f866d Filesystems:[{Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7284ff950b032e64d654b0acfc8eb152fd125b412977f0e54cab683340f403c0/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f2f56472fe4a6cd80dc8ec764bd26f5676f984d4abac9195e606f7b77ae1d3e/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dac0ff13d8ca56bec757c0fa7d01d7255ef736d2659fed84992e36c3f534da02/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a/volumes/kubernetes.io~projected/kube-api-access-hvq5c DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/166a115e5de2c1dbccbc5db7da0885d9bd0be7de55b06a775bc120b8b1ac8418/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c064fdf0dbba04a334dcb60c4ae08ff8ced20dde7540b2503943437534e6bd4d/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6/volumes/kubernetes.io~projected/kube-api-access-n25wg DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22bac4646e71cf5ee72bcf0d266008029d24a36f738cc29e69a7a5cf035f470c/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3824767d-0c5a-420e-9541-0ca551ee1e24/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:379 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ecccf676c96eef4a7e265553e6e99eb4058c21a4984efcf21d9f425b8077843/userdata/shm DeviceMajor:0 DeviceMinor:388 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~projected/kube-api-access-hhhxh DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d963d561ca7b1047ba8401a18917f6e0df0bfe5b71baeb431d276e75670f17c/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~projected/kube-api-access-v2d2q DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:375 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92e499a486b158b6007853855bb2503b83f073a15b37fd0382625097bb827819/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eead1d1a71ef79ffcb5a6e63d34cbf3da2e3261ae3505ae71eca55daaf485a62/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2/volumes/kubernetes.io~projected/kube-api-access-8s82w DeviceMajor:0 DeviceMinor:381 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d17c413475fb29445320726e612e9b052e31784c1281e3732ff0e0c5d36c59a/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~projected/kube-api-access-jn5dr DeviceMajor:0 DeviceMinor:380 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30b9f368-1f2b-41b8-a67a-6d17f4b9344c/volumes/kubernetes.io~projected/kube-api-access-4zj4q DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d5f7eda-1c08-4147-9fc8-241e0c982451/volumes/kubernetes.io~projected/kube-api-access-cwbxn DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ae28258-51a5-4e33-8040-39d8592006b8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volumes/kubernetes.io~projected/kube-api-access-x8rww DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/562612ccbfb1e40ce600a722a08dc3761a7d9eefce1f5b82438d5ec432e6ca4b/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~projected/kube-api-access-9962v DeviceMajor:0 DeviceMinor:260 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55e89fe3e687f542a384a773b510f27b59607f254bdcb24bec9ee5c00898e0da/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:378 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~projected/kube-api-access-49gqp DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a85e903-5cfa-4bb0-b902-d510337216c9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dceba17a3f1e34b731836ba6fb50d5aa2fa4f84c93849f7c00fe48bd06d2a9d9/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~projected/kube-api-access-thktd DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/168a3d5f-16fa-44c2-8123-bcf76bdad253/volumes/kubernetes.io~projected/kube-api-access-wbsdl DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9a4c7d5d-9276-4dce-8e0a-d7782b067792/volumes/kubernetes.io~projected/kube-api-access-8vzml DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0a539aa1f85b275b7fefef1d17b8caced37a43a2600b776cf9179203b10a1cd/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:200 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1b4b83378b045a9b2143a38a8d596aec4a6e8a35bf04e653ee63b5c225f9a2f/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66814c4de9447e1571930fdb20c3867410771c46116e316142607a8a8bb0fe32/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2da82773ebeff9c10c719f6a6e3d1bcd6fe4ba5a635f8c0bb3e323af6ed3c218/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1f2dabbc-40dc-4d65-b445-bcde680092ad/volumes/kubernetes.io~projected/kube-api-access-jd4pn DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:373 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b4b83c9cc89dc8a01bf188266c0fe664964175b508c0adf9b9092e97ff001f0/userdata/shm DeviceMajor:0 DeviceMinor:382 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eab012ae-9e57-45bb-8f50-9fe5f511a30c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a4c7d5d-9276-4dce-8e0a-d7782b067792/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:368 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3824767d-0c5a-420e-9541-0ca551ee1e24/volumes/kubernetes.io~projected/kube-api-access-nf4ht DeviceMajor:0 DeviceMinor:325 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c968f13c-daba-49a5-99a7-38063f221675/volumes/kubernetes.io~projected/kube-api-access-4vh67 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7431250c2191f92e57f287ae3ebd91dc0c6566a9e73c8c9b6fe9eeb3d568d650/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/208ff40c4ba58bbbd1c303d9837451ea37567b80ef280dbf88d53c175709b2c9/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cd8078a-c324-4fc8-b3d2-b631c6a2a724/volumes/kubernetes.io~projected/kube-api-access-knw45 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d804b66e-c2da-42b6-add0-c11761995171/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:374 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31c97247-cf8b-43b8-8fe4-4979ff070278/volumes/kubernetes.io~projected/kube-api-access-7cvrw DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/660bf9f5a55737482b05797181392636e59ca8d95332bd226ed263cee90fecd0/userdata/shm DeviceMajor:0 DeviceMinor:326 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c968f13c-daba-49a5-99a7-38063f221675/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0cd8078a-c324-4fc8-b3d2-b631c6a2a724/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/74280e02-7f3b-4a07-8bac-914610c5d04d/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b02c000-6d43-4adf-ba8b-4afd3e1186bc/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:375 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:22bac4646e71cf5 MacAddress:8e:6e:e7:08:73:7d Speed:10000 Mtu:8901} {Name:2da82773ebeff9c MacAddress:d2:e6:f4:5f:dc:2d Speed:10000 Mtu:8901} {Name:562612ccbfb1e40 MacAddress:b6:5a:d4:8e:9f:ed Speed:10000 Mtu:8901} {Name:5ecccf676c96eef MacAddress:ee:65:e5:a7:be:41 Speed:10000 Mtu:8901} {Name:660bf9f5a557374 MacAddress:42:2e:cb:5f:e4:8f Speed:10000 Mtu:8901} {Name:66814c4de9447e1 MacAddress:36:fa:be:df:67:e5 Speed:10000 Mtu:8901} {Name:6b4b83c9cc89dc8 MacAddress:ca:95:c9:fa:b2:b8 Speed:10000 Mtu:8901} {Name:7431250c2191f92 MacAddress:d2:c3:96:39:a9:84 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:df:bb:71:27 Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:ca:1a:bd:21:78 Speed:0 Mtu:8901} {Name:c064fdf0dbba04a MacAddress:32:5e:c1:46:f8:ea Speed:10000 Mtu:8901} {Name:dceba17a3f1e34b MacAddress:4e:3e:02:42:b7:e6 Speed:10000 Mtu:8901} {Name:e1b4b83378b045a MacAddress:82:68:3e:77:13:a5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:df:bb:71:27 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ea:da:ad:86:9f:87 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:71:90:e3:3f:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.707983 5420 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708030 5420 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708354 5420 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708378 5420 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-206.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708577 5420 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708590 5420 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708607 5420 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708633 5420 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708764 5420 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.708963 5420 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.709111 5420 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.709152 5420 kubelet.go:493] "Attempting to sync node with API server" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.709170 5420 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.709183 5420 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.709196 5420 kubelet.go:405] "Adding apiserver pod source" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.709208 5420 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.710993 5420 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.711008 5420 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 20:15:27.711795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.711639 5420 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 20:15:27.725870 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.725854 5420 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 20:15:27.725870 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.725868 5420 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 20:15:27.726015 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.725976 5420 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 20:15:27.726200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726186 5420 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 20:15:27.726264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726216 5420 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 20:15:27.726321 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726312 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 20:15:27.726365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726328 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 20:15:27.726365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726334 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 20:15:27.726365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726340 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 20:15:27.726365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726345 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 20:15:27.726365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726351 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 20:15:27.726365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726357 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 20:15:27.726365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726365 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 20:15:27.726700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726374 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 20:15:27.726700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726382 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 20:15:27.726700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726396 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 20:15:27.726700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726410 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 20:15:27.726700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726446 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 20:15:27.726700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.726455 5420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 20:15:27.730210 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.730193 5420 server.go:1263] "Started kubelet" Nov 02 20:15:27.730327 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.730261 5420 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 20:15:27.730430 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.730380 5420 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 20:15:27.730488 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.730448 5420 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 20:15:27.730782 ip-10-0-134-206 systemd[1]: Started Kubernetes Kubelet. Nov 02 20:15:27.730907 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.730849 5420 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 20:15:27.732306 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.731497 5420 server.go:317] "Adding debug handlers to kubelet server" Nov 02 20:15:27.734556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.733617 5420 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 20:15:27.734556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.733653 5420 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 20:15:27.734556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.733768 5420 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 20:15:27.734556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.733782 5420 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 20:15:27.734556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.733759 5420 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 20:09:17 +0000 UTC" deadline="2028-06-11 11:49:24.117250104 +0000 UTC" Nov 02 20:15:27.734556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.733795 5420 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14079h33m56.383458616s" Nov 02 20:15:27.734556 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:27.733769 5420 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-206.us-west-2.compute.internal\" not found" Nov 02 20:15:27.734556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.734254 5420 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737602 5420 factory.go:153] Registering CRI-O factory Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737625 5420 factory.go:223] Registration of the crio container factory successfully Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737696 5420 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737705 5420 factory.go:55] Registering systemd factory Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737712 5420 factory.go:223] Registration of the systemd container factory successfully Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737729 5420 factory.go:103] Registering Raw factory Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737749 5420 manager.go:1199] Started watching for new ooms in manager Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737942 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3" volumeName="kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.737993 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ae28258-51a5-4e33-8040-39d8592006b8" volumeName="kubernetes.io/secret/0ae28258-51a5-4e33-8040-39d8592006b8-agent-certs" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738016 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a85e903-5cfa-4bb0-b902-d510337216c9" volumeName="kubernetes.io/secret/2a85e903-5cfa-4bb0-b902-d510337216c9-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738032 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3824767d-0c5a-420e-9541-0ca551ee1e24" volumeName="kubernetes.io/projected/3824767d-0c5a-420e-9541-0ca551ee1e24-kube-api-access-nf4ht" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738046 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738066 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a85e903-5cfa-4bb0-b902-d510337216c9" volumeName="kubernetes.io/projected/2a85e903-5cfa-4bb0-b902-d510337216c9-kube-api-access-x8rww" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738079 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b02c000-6d43-4adf-ba8b-4afd3e1186bc" volumeName="kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738097 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="168a3d5f-16fa-44c2-8123-bcf76bdad253" volumeName="kubernetes.io/configmap/168a3d5f-16fa-44c2-8123-bcf76bdad253-iptables-alerter-script" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738116 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738135 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738147 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a85e903-5cfa-4bb0-b902-d510337216c9" volumeName="kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-script-lib" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738160 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e28c5b-ebf2-41aa-9be9-52e5e31aafa6" volumeName="kubernetes.io/projected/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-kube-api-access-n25wg" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738176 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3824767d-0c5a-420e-9541-0ca551ee1e24" volumeName="kubernetes.io/secret/3824767d-0c5a-420e-9541-0ca551ee1e24-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738192 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738215 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3" volumeName="kubernetes.io/empty-dir/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-textfile" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738227 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cd8078a-c324-4fc8-b3d2-b631c6a2a724" volumeName="kubernetes.io/secret/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-metrics-tls" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738246 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="168a3d5f-16fa-44c2-8123-bcf76bdad253" volumeName="kubernetes.io/projected/168a3d5f-16fa-44c2-8123-bcf76bdad253-kube-api-access-wbsdl" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738261 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738323 5420 manager.go:319] Starting recovery of all containers Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738412 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3" volumeName="kubernetes.io/projected/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-kube-api-access-9962v" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738450 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7c610f-2bc8-45db-b648-3d7b4d3cee2b" volumeName="kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-metrics-client-ca" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738479 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c968f13c-daba-49a5-99a7-38063f221675" volumeName="kubernetes.io/configmap/c968f13c-daba-49a5-99a7-38063f221675-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 20:15:27.737786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738504 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c968f13c-daba-49a5-99a7-38063f221675" volumeName="kubernetes.io/secret/c968f13c-daba-49a5-99a7-38063f221675-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738576 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b9f368-1f2b-41b8-a67a-6d17f4b9344c" volumeName="kubernetes.io/projected/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-kube-api-access-4zj4q" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738596 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/configmap/74280e02-7f3b-4a07-8bac-914610c5d04d-metrics-client-ca" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738615 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab012ae-9e57-45bb-8f50-9fe5f511a30c" volumeName="kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-tuned" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738630 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738700 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30b9f368-1f2b-41b8-a67a-6d17f4b9344c" volumeName="kubernetes.io/configmap/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-serviceca" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738716 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ae28258-51a5-4e33-8040-39d8592006b8" volumeName="kubernetes.io/configmap/0ae28258-51a5-4e33-8040-39d8592006b8-konnectivity-ca" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738774 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738825 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c97247-cf8b-43b8-8fe4-4979ff070278" volumeName="kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738841 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e28c5b-ebf2-41aa-9be9-52e5e31aafa6" volumeName="kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-daemon-config" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738880 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3" volumeName="kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-tls" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738897 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c968f13c-daba-49a5-99a7-38063f221675" volumeName="kubernetes.io/empty-dir/c968f13c-daba-49a5-99a7-38063f221675-data-volume" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.738982 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c968f13c-daba-49a5-99a7-38063f221675" volumeName="kubernetes.io/projected/c968f13c-daba-49a5-99a7-38063f221675-kube-api-access-4vh67" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739005 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739102 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cd8078a-c324-4fc8-b3d2-b631c6a2a724" volumeName="kubernetes.io/configmap/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-config-volume" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739114 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f2dabbc-40dc-4d65-b445-bcde680092ad" volumeName="kubernetes.io/empty-dir/1f2dabbc-40dc-4d65-b445-bcde680092ad-tmp-dir" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739125 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739141 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c97247-cf8b-43b8-8fe4-4979ff070278" volumeName="kubernetes.io/projected/31c97247-cf8b-43b8-8fe4-4979ff070278-kube-api-access-7cvrw" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739153 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aaa4ca9-71ad-4f88-898f-c6bc2799f20a" volumeName="kubernetes.io/secret/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-cert" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739169 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3" volumeName="kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-metrics-client-ca" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739181 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7c610f-2bc8-45db-b648-3d7b4d3cee2b" volumeName="kubernetes.io/empty-dir/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-volume-directive-shadow" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739193 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cd8078a-c324-4fc8-b3d2-b631c6a2a724" volumeName="kubernetes.io/empty-dir/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-tmp-dir" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739208 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cd8078a-c324-4fc8-b3d2-b631c6a2a724" volumeName="kubernetes.io/projected/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-kube-api-access-knw45" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739252 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739271 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739286 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739298 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75ef20c6-f775-4fc4-96e3-8bd5f4a983f2" volumeName="kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739315 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3" volumeName="kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739328 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739346 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a85e903-5cfa-4bb0-b902-d510337216c9" volumeName="kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-env-overrides" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739393 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-grpc-tls" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739405 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7c610f-2bc8-45db-b648-3d7b4d3cee2b" volumeName="kubernetes.io/projected/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-api-access-v2d2q" seLinuxMountContext="" Nov 02 20:15:27.739929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739758 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d804b66e-c2da-42b6-add0-c11761995171" volumeName="kubernetes.io/configmap/d804b66e-c2da-42b6-add0-c11761995171-metrics-client-ca" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.739809 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d804b66e-c2da-42b6-add0-c11761995171" volumeName="kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741254 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741286 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741302 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34e28c5b-ebf2-41aa-9be9-52e5e31aafa6" volumeName="kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cni-binary-copy" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741314 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/projected/74280e02-7f3b-4a07-8bac-914610c5d04d-kube-api-access-hhhxh" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741325 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-tls" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741338 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d804b66e-c2da-42b6-add0-c11761995171" volumeName="kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741350 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741362 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741389 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7c610f-2bc8-45db-b648-3d7b4d3cee2b" volumeName="kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741401 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab012ae-9e57-45bb-8f50-9fe5f511a30c" volumeName="kubernetes.io/projected/eab012ae-9e57-45bb-8f50-9fe5f511a30c-kube-api-access-49gqp" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741413 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f2dabbc-40dc-4d65-b445-bcde680092ad" volumeName="kubernetes.io/projected/1f2dabbc-40dc-4d65-b445-bcde680092ad-kube-api-access-jd4pn" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741424 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741435 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741447 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a4c7d5d-9276-4dce-8e0a-d7782b067792" volumeName="kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741458 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d804b66e-c2da-42b6-add0-c11761995171" volumeName="kubernetes.io/projected/d804b66e-c2da-42b6-add0-c11761995171-kube-api-access-thktd" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741472 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741485 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7c610f-2bc8-45db-b648-3d7b4d3cee2b" volumeName="kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741498 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d5f7eda-1c08-4147-9fc8-241e0c982451" volumeName="kubernetes.io/projected/1d5f7eda-1c08-4147-9fc8-241e0c982451-kube-api-access-cwbxn" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741511 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741522 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741534 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741547 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a85e903-5cfa-4bb0-b902-d510337216c9" volumeName="kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-config" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741560 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aaa4ca9-71ad-4f88-898f-c6bc2799f20a" volumeName="kubernetes.io/projected/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-kube-api-access-hvq5c" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741573 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb7c610f-2bc8-45db-b648-3d7b4d3cee2b" volumeName="kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741586 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab012ae-9e57-45bb-8f50-9fe5f511a30c" volumeName="kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-tmp" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741598 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dbeb5e-76a3-4e56-9843-60aaeb743e39" volumeName="kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741610 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c97247-cf8b-43b8-8fe4-4979ff070278" volumeName="kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-binary-copy" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741623 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31c97247-cf8b-43b8-8fe4-4979ff070278" volumeName="kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741635 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="74280e02-7f3b-4a07-8bac-914610c5d04d" volumeName="kubernetes.io/empty-dir/74280e02-7f3b-4a07-8bac-914610c5d04d-tmp" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741647 5420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a4c7d5d-9276-4dce-8e0a-d7782b067792" volumeName="kubernetes.io/projected/9a4c7d5d-9276-4dce-8e0a-d7782b067792-kube-api-access-8vzml" seLinuxMountContext="" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741659 5420 reconstruct.go:97] "Volume reconstruction finished" Nov 02 20:15:27.742012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.741687 5420 reconciler.go:29] "Reconciler: start to sync state" Nov 02 20:15:27.749715 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.749690 5420 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 20:15:27.766427 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.766400 5420 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 20:15:27.766427 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.766427 5420 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 20:15:27.766837 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.766446 5420 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 20:15:27.766837 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:27.766489 5420 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 20:15:27.790205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.790135 5420 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="9716d8a6f31b50398651a89eec0dc6f6df15396258922f200368e2c6c39c9e0d" exitCode=0 Nov 02 20:15:27.790205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.790200 5420 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="7a6097dafd9e22842d6cd1be6c19373be59e0fe5855bddb3a6b810a41f14d24b" exitCode=0 Nov 02 20:15:27.790365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.790210 5420 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="cb75ae3b26a8a627b594d05a001444c08ae17e40a0a1cfb8f96a0c0131f6dc02" exitCode=0 Nov 02 20:15:27.790365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.790223 5420 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="07e33c609da4e9e4930dced4365684b31dfd92364fee6681acb1f66d4f6494d0" exitCode=0 Nov 02 20:15:27.790365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.790230 5420 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="fd9eb2c17817046edf92586052f5f6dba7cf3dfc588a6a76cdf3732e44ffc44e" exitCode=0 Nov 02 20:15:27.790365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.790238 5420 generic.go:356] "Generic (PLEG): container finished" podID="31c97247-cf8b-43b8-8fe4-4979ff070278" containerID="6e67e70f5ca11f0b29e18e3957f46748bd82c08fd1906b4527cad1e2c3b1f95b" exitCode=0 Nov 02 20:15:27.810886 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.810853 5420 generic.go:356] "Generic (PLEG): container finished" podID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" exitCode=0 Nov 02 20:15:27.815042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.814988 5420 generic.go:356] "Generic (PLEG): container finished" podID="aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3" containerID="ba462a0249ea84347053453e9410fccd4ee8ddf7f254a50968e54bb591b862a8" exitCode=0 Nov 02 20:15:27.823134 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.823104 5420 generic.go:356] "Generic (PLEG): container finished" podID="506cfda16cd111388fffa21903437d2c" containerID="a2854149307d33177ecbe4a5857512779a487da6cbce62e0f567d52f42f95659" exitCode=0 Nov 02 20:15:27.834382 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:27.834363 5420 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-206.us-west-2.compute.internal\" not found" Nov 02 20:15:27.843443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.843424 5420 manager.go:324] Recovery completed Nov 02 20:15:27.866170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.866146 5420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:15:27.867447 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:27.867422 5420 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 20:15:27.868746 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.868728 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:15:27.868746 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.868758 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:15:27.868891 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.868774 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:15:27.871404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871386 5420 cpu_manager.go:226] "Starting" policy="none" Nov 02 20:15:27.871404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871401 5420 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 20:15:27.871541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871402 5420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:15:27.871541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871423 5420 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 20:15:27.871624 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871568 5420 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 20:15:27.871624 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871582 5420 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 20:15:27.871624 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871607 5420 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 20:15:27.871624 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871615 5420 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 20:15:27.871624 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871623 5420 policy_none.go:50] "Start" Nov 02 20:15:27.871816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871634 5420 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 20:15:27.871816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871647 5420 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 20:15:27.871912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871836 5420 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 20:15:27.871912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871848 5420 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 20:15:27.871912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.871856 5420 policy_none.go:44] "Start" Nov 02 20:15:27.872354 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.872335 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:15:27.872434 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.872365 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:15:27.872434 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.872379 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:15:27.878212 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.878186 5420 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 20:15:27.881234 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.881214 5420 manager.go:341] "Starting Device Plugin manager" Nov 02 20:15:27.881322 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:27.881242 5420 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 20:15:27.881322 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.881251 5420 server.go:87] "Starting device plugin registration server" Nov 02 20:15:27.881507 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.881494 5420 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 20:15:27.881557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.881510 5420 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 20:15:27.882738 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.882719 5420 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 20:15:27.882843 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.882810 5420 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 20:15:27.882936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.882852 5420 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 20:15:27.882936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.882858 5420 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 20:15:27.883751 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.882908 5420 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T20:15:27.88283937Z","UUID":"1aa2a859-4c91-4f05-b3f2-c84106f7ffc4","Handler":null,"Name":"","Endpoint":""} Nov 02 20:15:27.884550 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:27.884530 5420 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 20:15:27.884646 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:27.884569 5420 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-206.us-west-2.compute.internal\" not found" Nov 02 20:15:27.885074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.885051 5420 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 20:15:27.885184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.885094 5420 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 20:15:27.907281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.907259 5420 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:15:27.907378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.907282 5420 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:15:27.908265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.908240 5420 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:15:27.908265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.908246 5420 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 20:15:27.981890 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.981841 5420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 20:15:27.983220 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.983197 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 20:15:27.983344 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.983244 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 20:15:27.983344 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.983260 5420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-206.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 20:15:27.983427 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.983394 5420 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:27.999180 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:27.999149 5420 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 20:15:28.068387 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.068295 5420 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal"] Nov 02 20:15:28.069161 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.069136 5420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.069281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.069139 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"d51646e0acf14788341d5bd3c94d4017","Type":"ContainerStarted","Data":"4271b54f71378a3ab00f8163c7889b42bb6e3f7b40aac9bce800c743cdc47d4c"} Nov 02 20:15:28.069281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.069188 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"d51646e0acf14788341d5bd3c94d4017","Type":"ContainerStarted","Data":"eead1d1a71ef79ffcb5a6e63d34cbf3da2e3261ae3505ae71eca55daaf485a62"} Nov 02 20:15:28.069397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.069281 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"506cfda16cd111388fffa21903437d2c","Type":"ContainerStarted","Data":"dc7a8404001189b5b5622f568d73481904e61899ce4055de6662f2968861eae9"} Nov 02 20:15:28.069397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.069302 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"506cfda16cd111388fffa21903437d2c","Type":"ContainerDied","Data":"a2854149307d33177ecbe4a5857512779a487da6cbce62e0f567d52f42f95659"} Nov 02 20:15:28.069397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.069322 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" event={"ID":"506cfda16cd111388fffa21903437d2c","Type":"ContainerStarted","Data":"0d17c413475fb29445320726e612e9b052e31784c1281e3732ff0e0c5d36c59a"} Nov 02 20:15:28.069397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.069345 5420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.085926 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.085906 5420 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.086039 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.085967 5420 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.128181 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.128157 5420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:15:28.128325 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:28.128211 5420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.128786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.128762 5420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:15:28.128890 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:28.128808 5420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.209318 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.209287 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.209318 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.209320 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.209532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.209336 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d51646e0acf14788341d5bd3c94d4017-config\") pod \"kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"d51646e0acf14788341d5bd3c94d4017\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.310098 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.310068 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.310098 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.310103 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.310276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.310122 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d51646e0acf14788341d5bd3c94d4017-config\") pod \"kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"d51646e0acf14788341d5bd3c94d4017\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.310276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.310178 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.310276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.310206 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d51646e0acf14788341d5bd3c94d4017-config\") pod \"kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"d51646e0acf14788341d5bd3c94d4017\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.310276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.310178 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/506cfda16cd111388fffa21903437d2c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" (UID: \"506cfda16cd111388fffa21903437d2c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.709783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.709755 5420 apiserver.go:52] "Watching apiserver" Nov 02 20:15:28.832650 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.832626 5420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.832814 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.832784 5420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.852034 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.852010 5420 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 20:15:28.852613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.852594 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-8plx8","kube-system/konnectivity-agent-wnw5s","openshift-ingress-canary/ingress-canary-7rth8","openshift-network-operator/iptables-alerter-2l8lk","openshift-cluster-node-tuning-operator/tuned-898kr","kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr","openshift-network-diagnostics/network-check-target-89w6h","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-ndz76","openshift-image-registry/node-ca-8hm6m","openshift-insights/insights-runtime-extractor-x5lv8","openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh","openshift-dns/dns-default-gjf9l","openshift-multus/multus-additional-cni-plugins-6ks29","openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p","openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal","openshift-dns/node-resolver-6zxch","openshift-monitoring/node-exporter-6tlrn","openshift-multus/multus-4lc2l","kube-system/global-pull-secret-syncer-s49d6","openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn"] Nov 02 20:15:28.890188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.890161 5420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:15:28.890298 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:28.890203 5420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.890869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.890858 5420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 20:15:28.890933 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:28.890887 5420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-206.us-west-2.compute.internal" Nov 02 20:15:28.902080 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.902062 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 20:15:28.902154 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.902114 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-t6dvk\"" Nov 02 20:15:28.902823 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.902807 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7m5tr\"" Nov 02 20:15:28.902908 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.902853 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxslf\"" Nov 02 20:15:28.903134 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903120 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 20:15:28.903423 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903402 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 20:15:28.903423 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903425 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xxqx9\"" Nov 02 20:15:28.903557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903498 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 20:15:28.903611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903587 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 20:15:28.903663 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903650 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 20:15:28.903753 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903663 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 20:15:28.903753 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903703 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 20:15:28.903753 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903712 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 20:15:28.903753 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903709 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:15:28.903957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903756 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vjrtf\"" Nov 02 20:15:28.903957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903764 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 20:15:28.903957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903894 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 20:15:28.903957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903895 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qfmz2\"" Nov 02 20:15:28.903957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903935 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 20:15:28.903957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.903959 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 20:15:28.904200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904162 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 20:15:28.904200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904190 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 20:15:28.904200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904192 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 20:15:28.904374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904343 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 20:15:28.904374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904350 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ldshw\"" Nov 02 20:15:28.904468 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904382 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 20:15:28.904468 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904402 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 20:15:28.904468 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904402 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4wc8w\"" Nov 02 20:15:28.904650 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904636 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 20:15:28.904743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904698 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 20:15:28.904743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904711 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 20:15:28.904818 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904787 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s29pz\"" Nov 02 20:15:28.904922 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904907 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 20:15:28.904979 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904930 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5vs7j\"" Nov 02 20:15:28.904979 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.904949 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 20:15:28.905107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.905095 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 20:15:28.905156 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.905094 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 20:15:28.905156 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.905095 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:15:28.936326 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.936307 5420 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 20:15:28.961591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.961489 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 20:15:28.961591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.961509 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 20:15:28.962143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962041 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 02 20:15:28.962143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962076 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 02 20:15:28.962143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962111 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 20:15:28.962143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962116 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 20:15:28.962143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962125 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 20:15:28.962553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962533 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962556 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962558 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962571 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962558 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5ep5s2cc5pb2s\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962558 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962600 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962608 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962623 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dfwqd\"" Nov 02 20:15:28.962644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962631 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962657 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962664 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bvgts\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962626 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4tnzz\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962889 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aa2ub3sv54fn3\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962910 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-2nsqw\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962913 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k9j84\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962936 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962932 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962980 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-82tkg\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962991 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962999 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-7bq62\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.962981 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jvjxk\"" Nov 02 20:15:28.963025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963027 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-74k7h\"" Nov 02 20:15:28.963401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963039 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 20:15:28.963401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963061 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 20:15:28.963401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963061 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d6nbh\"" Nov 02 20:15:28.963401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963080 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 20:15:28.963401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963096 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 20:15:28.963401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963097 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 20:15:28.963401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963121 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 20:15:28.963401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:28.963131 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kz985\"" Nov 02 20:15:29.003031 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003004 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 20:15:29.003155 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003138 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 20:15:29.003217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003154 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 20:15:29.003217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003203 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 20:15:29.003217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003214 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 20:15:29.003364 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003222 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 20:15:29.003364 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003225 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 20:15:29.003364 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003218 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 20:15:29.003508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003380 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 20:15:29.003508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003409 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 20:15:29.003508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003430 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 20:15:29.003718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003575 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 20:15:29.003718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003583 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 20:15:29.003718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003642 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 20:15:29.003718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003650 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 20:15:29.003878 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003799 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 20:15:29.003878 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.003831 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 20:15:29.009688 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.009638 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 20:15:29.010312 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.010247 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 20:15:29.014465 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014446 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c968f13c-daba-49a5-99a7-38063f221675-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.014540 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014473 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.014540 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014490 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.014540 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014514 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014539 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014562 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2d2q\" (UniqueName: \"kubernetes.io/projected/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-api-access-v2d2q\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014599 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4ht\" (UniqueName: \"kubernetes.io/projected/3824767d-0c5a-420e-9541-0ca551ee1e24-kube-api-access-nf4ht\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014622 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014637 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014661 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-system-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014702 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-kubernetes\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014727 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvq5c\" (UniqueName: \"kubernetes.io/projected/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-kube-api-access-hvq5c\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014751 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-config-volume\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014780 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014805 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74280e02-7f3b-4a07-8bac-914610c5d04d-metrics-client-ca\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014809 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c968f13c-daba-49a5-99a7-38063f221675-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014821 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014839 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014853 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.014915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014865 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.014921 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-original-pull-secret\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015027 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-config-volume\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015079 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74280e02-7f3b-4a07-8bac-914610c5d04d-metrics-client-ca\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015078 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015114 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015134 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015141 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015145 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-slash\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015178 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-kubelet-config\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015194 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015200 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vh67\" (UniqueName: \"kubernetes.io/projected/c968f13c-daba-49a5-99a7-38063f221675-kube-api-access-4vh67\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015245 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-root\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015275 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015299 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015324 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cni-binary-copy\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015347 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-multus-certs\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015372 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015422 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c968f13c-daba-49a5-99a7-38063f221675-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015458 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015428 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015469 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015483 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-host\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015507 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-systemd\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015521 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cni-binary-copy\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015532 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-metrics-tls\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015555 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.015587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015577 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ae28258-51a5-4e33-8040-39d8592006b8-konnectivity-ca\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015604 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c968f13c-daba-49a5-99a7-38063f221675-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015635 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-sys\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015721 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015757 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-metrics-tls\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015774 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015782 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0ae28258-51a5-4e33-8040-39d8592006b8-konnectivity-ca\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015814 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gqp\" (UniqueName: \"kubernetes.io/projected/eab012ae-9e57-45bb-8f50-9fe5f511a30c-kube-api-access-49gqp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015820 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015858 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vzml\" (UniqueName: \"kubernetes.io/projected/9a4c7d5d-9276-4dce-8e0a-d7782b067792-kube-api-access-8vzml\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015885 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.015946 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016004 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016019 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-tmp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016039 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-systemd-units\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016056 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn5dr\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016099 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-tmp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016132 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-dbus\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016161 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thktd\" (UniqueName: \"kubernetes.io/projected/d804b66e-c2da-42b6-add0-c11761995171-kube-api-access-thktd\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016190 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9962v\" (UniqueName: \"kubernetes.io/projected/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-kube-api-access-9962v\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016217 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c968f13c-daba-49a5-99a7-38063f221675-data-volume\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016287 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c968f13c-daba-49a5-99a7-38063f221675-data-volume\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016319 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016345 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ae28258-51a5-4e33-8040-39d8592006b8-agent-certs\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016370 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016380 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016395 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-hostroot\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016422 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cvrw\" (UniqueName: \"kubernetes.io/projected/31c97247-cf8b-43b8-8fe4-4979ff070278-kube-api-access-7cvrw\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016456 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2dabbc-40dc-4d65-b445-bcde680092ad-hosts-file\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016477 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-cert\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016508 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016512 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016545 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016605 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016609 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0ae28258-51a5-4e33-8040-39d8592006b8-agent-certs\") pod \"konnectivity-agent-wnw5s\" (UID: \"0ae28258-51a5-4e33-8040-39d8592006b8\") " pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016649 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-device-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.016667 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016712 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-cert\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016713 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-var-lib-kubelet\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016726 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016751 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cnibin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016778 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-sys-fs\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016784 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016804 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbxn\" (UniqueName: \"kubernetes.io/projected/1d5f7eda-1c08-4147-9fc8-241e0c982451-kube-api-access-cwbxn\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016835 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-tuned\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016853 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-textfile\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016868 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-tls\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016882 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016896 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhxh\" (UniqueName: \"kubernetes.io/projected/74280e02-7f3b-4a07-8bac-914610c5d04d-kube-api-access-hhhxh\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016954 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-textfile\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016953 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-tuned\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016973 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.016997 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017025 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017055 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017092 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-tls\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017095 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-bin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017137 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017154 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74280e02-7f3b-4a07-8bac-914610c5d04d-tmp\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017169 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25wg\" (UniqueName: \"kubernetes.io/projected/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-kube-api-access-n25wg\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017194 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017208 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zj4q\" (UniqueName: \"kubernetes.io/projected/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-kube-api-access-4zj4q\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017224 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017239 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-env-overrides\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017240 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74280e02-7f3b-4a07-8bac-914610c5d04d-tmp\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017254 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-tmp-dir\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017293 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017296 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017330 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017335 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-tmp-dir\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017356 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017441 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-env-overrides\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017468 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-multus\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017538 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-cnibin\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017560 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017570 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017588 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017589 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017688 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-metrics-client-ca\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017722 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017726 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d804b66e-c2da-42b6-add0-c11761995171-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017751 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-kubelet\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017753 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017799 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017843 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-netd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017874 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-metrics-client-ca\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017890 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-netns\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017914 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-wtmp\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.017916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017936 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-conf-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017954 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/168a3d5f-16fa-44c2-8123-bcf76bdad253-host-slash\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.017976 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2dabbc-40dc-4d65-b445-bcde680092ad-tmp-dir\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018004 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-socket-dir-parent\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018022 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysconfig\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018041 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-etc-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018095 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f2dabbc-40dc-4d65-b445-bcde680092ad-tmp-dir\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018107 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-script-lib\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018135 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/168a3d5f-16fa-44c2-8123-bcf76bdad253-iptables-alerter-script\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018155 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4pn\" (UniqueName: \"kubernetes.io/projected/1f2dabbc-40dc-4d65-b445-bcde680092ad-kube-api-access-jd4pn\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018217 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-netns\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018259 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-node-log\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018288 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018302 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/168a3d5f-16fa-44c2-8123-bcf76bdad253-iptables-alerter-script\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018391 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-system-cni-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018418 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-script-lib\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018440 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018465 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4c7d5d-9276-4dce-8e0a-d7782b067792-metrics-certs\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018483 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c968f13c-daba-49a5-99a7-38063f221675-crio-socket\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018507 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbsdl\" (UniqueName: \"kubernetes.io/projected/168a3d5f-16fa-44c2-8123-bcf76bdad253-kube-api-access-wbsdl\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018528 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-kubelet\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018545 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018715 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018764 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018784 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018799 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-run\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018816 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-lib-modules\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018856 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-ovn\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018885 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-log-socket\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018889 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018888 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018918 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knw45\" (UniqueName: \"kubernetes.io/projected/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-kube-api-access-knw45\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018946 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3824767d-0c5a-420e-9541-0ca551ee1e24-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.018990 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019011 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-etc-kubernetes\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019082 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-conf\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019117 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a85e903-5cfa-4bb0-b902-d510337216c9-ovn-node-metrics-cert\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019149 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019170 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3824767d-0c5a-420e-9541-0ca551ee1e24-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019210 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-serviceca\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019248 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-registration-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019275 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-sys\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019300 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-config\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019320 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/31c97247-cf8b-43b8-8fe4-4979ff070278-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019329 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d804b66e-c2da-42b6-add0-c11761995171-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019346 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019356 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a85e903-5cfa-4bb0-b902-d510337216c9-ovn-node-metrics-cert\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019468 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019496 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-systemd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019501 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019518 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d804b66e-c2da-42b6-add0-c11761995171-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019553 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-grpc-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019581 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019589 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2a85e903-5cfa-4bb0-b902-d510337216c9-ovnkube-config\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019608 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-k8s-cni-cncf-io\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019634 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-socket-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019659 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-etc-selinux\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019731 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019735 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-host\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.019739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019758 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019777 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-bin\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019820 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019828 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-serviceca\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019830 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74280e02-7f3b-4a07-8bac-914610c5d04d-secret-grpc-tls\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019908 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019927 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-daemon-config\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019941 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-modprobe-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019958 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019974 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-var-lib-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019990 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.020005 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8rww\" (UniqueName: \"kubernetes.io/projected/2a85e903-5cfa-4bb0-b902-d510337216c9-kube-api-access-x8rww\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.020064 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-daemon-config\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.020173 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.019946 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.021265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.020308 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.070190 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.070171 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vh67\" (UniqueName: \"kubernetes.io/projected/c968f13c-daba-49a5-99a7-38063f221675-kube-api-access-4vh67\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.072293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.072185 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvq5c\" (UniqueName: \"kubernetes.io/projected/7aaa4ca9-71ad-4f88-898f-c6bc2799f20a-kube-api-access-hvq5c\") pod \"ingress-canary-7rth8\" (UID: \"7aaa4ca9-71ad-4f88-898f-c6bc2799f20a\") " pod="openshift-ingress-canary/ingress-canary-7rth8" Nov 02 20:15:29.072293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.072271 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2d2q\" (UniqueName: \"kubernetes.io/projected/bb7c610f-2bc8-45db-b648-3d7b4d3cee2b-kube-api-access-v2d2q\") pod \"kube-state-metrics-5dc8d7d89b-g29mn\" (UID: \"bb7c610f-2bc8-45db-b648-3d7b4d3cee2b\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-g29mn" Nov 02 20:15:29.072293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.072276 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4ht\" (UniqueName: \"kubernetes.io/projected/3824767d-0c5a-420e-9541-0ca551ee1e24-kube-api-access-nf4ht\") pod \"monitoring-plugin-77f4f76594-ctj2p\" (UID: \"3824767d-0c5a-420e-9541-0ca551ee1e24\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:29.072484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.072273 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9962v\" (UniqueName: \"kubernetes.io/projected/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-kube-api-access-9962v\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.072484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.072316 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gqp\" (UniqueName: \"kubernetes.io/projected/eab012ae-9e57-45bb-8f50-9fe5f511a30c-kube-api-access-49gqp\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.072484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.072463 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn5dr\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr\") pod \"prometheus-k8s-0\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:29.072484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.072470 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thktd\" (UniqueName: \"kubernetes.io/projected/d804b66e-c2da-42b6-add0-c11761995171-kube-api-access-thktd\") pod \"openshift-state-metrics-7bd7489957-8dfdh\" (UID: \"d804b66e-c2da-42b6-add0-c11761995171\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dfdh" Nov 02 20:15:29.072759 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.072744 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vzml\" (UniqueName: \"kubernetes.io/projected/9a4c7d5d-9276-4dce-8e0a-d7782b067792-kube-api-access-8vzml\") pod \"network-metrics-daemon-ndz76\" (UID: \"9a4c7d5d-9276-4dce-8e0a-d7782b067792\") " pod="openshift-multus/network-metrics-daemon-ndz76" Nov 02 20:15:29.077855 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.077834 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cvrw\" (UniqueName: \"kubernetes.io/projected/31c97247-cf8b-43b8-8fe4-4979ff070278-kube-api-access-7cvrw\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.078183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.078160 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbxn\" (UniqueName: \"kubernetes.io/projected/1d5f7eda-1c08-4147-9fc8-241e0c982451-kube-api-access-cwbxn\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.078528 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.078503 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4pn\" (UniqueName: \"kubernetes.io/projected/1f2dabbc-40dc-4d65-b445-bcde680092ad-kube-api-access-jd4pn\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:15:29.078613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.078533 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25wg\" (UniqueName: \"kubernetes.io/projected/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-kube-api-access-n25wg\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.078613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.078562 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zj4q\" (UniqueName: \"kubernetes.io/projected/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-kube-api-access-4zj4q\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:15:29.078613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.078598 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbsdl\" (UniqueName: \"kubernetes.io/projected/168a3d5f-16fa-44c2-8123-bcf76bdad253-kube-api-access-wbsdl\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:15:29.078745 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.078731 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhxh\" (UniqueName: \"kubernetes.io/projected/74280e02-7f3b-4a07-8bac-914610c5d04d-kube-api-access-hhhxh\") pod \"thanos-querier-7bb5949b9c-jjt6t\" (UID: \"74280e02-7f3b-4a07-8bac-914610c5d04d\") " pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:29.078782 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.078769 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8rww\" (UniqueName: \"kubernetes.io/projected/2a85e903-5cfa-4bb0-b902-d510337216c9-kube-api-access-x8rww\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.078869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.078854 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s82w\" (UniqueName: \"kubernetes.io/projected/75ef20c6-f775-4fc4-96e3-8bd5f4a983f2-kube-api-access-8s82w\") pod \"network-check-target-89w6h\" (UID: \"75ef20c6-f775-4fc4-96e3-8bd5f4a983f2\") " pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:15:29.083115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.083098 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knw45\" (UniqueName: \"kubernetes.io/projected/0cd8078a-c324-4fc8-b3d2-b631c6a2a724-kube-api-access-knw45\") pod \"dns-default-gjf9l\" (UID: \"0cd8078a-c324-4fc8-b3d2-b631c6a2a724\") " pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:29.120601 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120572 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-systemd-units\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.120694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120606 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-dbus\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:29.120694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120642 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-hostroot\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.120694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120661 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2dabbc-40dc-4d65-b445-bcde680092ad-hosts-file\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:15:29.120694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120678 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-systemd-units\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120695 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-device-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120703 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-dbus\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120718 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-hostroot\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120717 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-var-lib-kubelet\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120741 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cnibin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120762 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-sys-fs\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120771 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f2dabbc-40dc-4d65-b445-bcde680092ad-hosts-file\") pod \"node-resolver-6zxch\" (UID: \"1f2dabbc-40dc-4d65-b445-bcde680092ad\") " pod="openshift-dns/node-resolver-6zxch" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120788 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120788 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-device-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120822 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-cnibin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120827 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-sys-fs\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120829 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-bin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120854 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-bin\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120860 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120830 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-var-lib-kubelet\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.120874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120830 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120898 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120927 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120932 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-multus\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120941 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-kubelet-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120955 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-cnibin\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120976 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-cni-multus\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.120980 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-kubelet\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121001 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121020 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-netd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121002 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-kubelet\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121023 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-cnibin\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121037 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-netns\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121041 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121055 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-netd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121065 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-run-netns\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121069 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-wtmp\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121098 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-conf-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121122 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/168a3d5f-16fa-44c2-8123-bcf76bdad253-host-slash\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121133 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-node-exporter-wtmp\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121145 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-socket-dir-parent\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121154 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-conf-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121170 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/168a3d5f-16fa-44c2-8123-bcf76bdad253-host-slash\") pod \"iptables-alerter-2l8lk\" (UID: \"168a3d5f-16fa-44c2-8123-bcf76bdad253\") " pod="openshift-network-operator/iptables-alerter-2l8lk" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121174 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysconfig\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121182 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-socket-dir-parent\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121196 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-etc-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121200 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysconfig\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121224 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-netns\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121247 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-node-log\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121248 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-etc-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121270 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-system-cni-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121280 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-netns\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121294 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121311 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-node-log\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121315 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c968f13c-daba-49a5-99a7-38063f221675-crio-socket\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121324 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/31c97247-cf8b-43b8-8fe4-4979ff070278-system-cni-dir\") pod \"multus-additional-cni-plugins-6ks29\" (UID: \"31c97247-cf8b-43b8-8fe4-4979ff070278\") " pod="openshift-multus/multus-additional-cni-plugins-6ks29" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121340 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-kubelet\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121365 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-run\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121369 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c968f13c-daba-49a5-99a7-38063f221675-crio-socket\") pod \"insights-runtime-extractor-x5lv8\" (UID: \"c968f13c-daba-49a5-99a7-38063f221675\") " pod="openshift-insights/insights-runtime-extractor-x5lv8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121388 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-var-lib-kubelet\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121388 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-lib-modules\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121412 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-run\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121430 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-ovn\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121437 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121452 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-log-socket\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121478 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-ovn\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121485 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-etc-kubernetes\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121499 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-lib-modules\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121509 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-conf\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121511 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-log-socket\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121518 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-etc-kubernetes\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121535 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-registration-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121562 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-sys\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121590 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-sysctl-conf\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121603 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-registration-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121620 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-sys\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121623 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-systemd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.121597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121638 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-run-systemd\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121656 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-k8s-cni-cncf-io\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121698 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-k8s-cni-cncf-io\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121701 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-socket-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121727 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-etc-selinux\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121740 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-host\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121753 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-bin\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121762 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-socket-dir\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121774 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-modprobe-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121790 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-var-lib-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121800 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-host\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121804 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-cni-bin\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121788 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1d5f7eda-1c08-4147-9fc8-241e0c982451-etc-selinux\") pod \"aws-ebs-csi-driver-node-scpzr\" (UID: \"1d5f7eda-1c08-4147-9fc8-241e0c982451\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-scpzr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121805 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121835 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-var-lib-openvswitch\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121847 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-system-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121860 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-modprobe-d\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121867 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121870 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-kubernetes\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121902 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121908 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-kubernetes\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121914 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-system-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121924 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-slash\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121944 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-kubelet-config\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121955 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a85e903-5cfa-4bb0-b902-d510337216c9-host-slash\") pod \"ovnkube-node-8plx8\" (UID: \"2a85e903-5cfa-4bb0-b902-d510337216c9\") " pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121963 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-root\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121964 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-multus-cni-dir\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121992 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-root\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.121994 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6b02c000-6d43-4adf-ba8b-4afd3e1186bc-kubelet-config\") pod \"global-pull-secret-syncer-s49d6\" (UID: \"6b02c000-6d43-4adf-ba8b-4afd3e1186bc\") " pod="kube-system/global-pull-secret-syncer-s49d6" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.122020 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-multus-certs\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.122038 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-host\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.122054 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-systemd\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.122069 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-sys\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.122075 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30b9f368-1f2b-41b8-a67a-6d17f4b9344c-host\") pod \"node-ca-8hm6m\" (UID: \"30b9f368-1f2b-41b8-a67a-6d17f4b9344c\") " pod="openshift-image-registry/node-ca-8hm6m" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.122055 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34e28c5b-ebf2-41aa-9be9-52e5e31aafa6-host-run-multus-certs\") pod \"multus-4lc2l\" (UID: \"34e28c5b-ebf2-41aa-9be9-52e5e31aafa6\") " pod="openshift-multus/multus-4lc2l" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.122113 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3-sys\") pod \"node-exporter-6tlrn\" (UID: \"aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3\") " pod="openshift-monitoring/node-exporter-6tlrn" Nov 02 20:15:29.123188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.122115 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eab012ae-9e57-45bb-8f50-9fe5f511a30c-etc-systemd\") pod \"tuned-898kr\" (UID: \"eab012ae-9e57-45bb-8f50-9fe5f511a30c\") " pod="openshift-cluster-node-tuning-operator/tuned-898kr" Nov 02 20:15:29.165596 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:29.165582 5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:15:30.841718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:30.841665 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerStarted","Data":"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b"} Nov 02 20:15:30.842082 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:30.841785 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerStarted","Data":"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d"} Nov 02 20:15:30.842082 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:30.841813 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerStarted","Data":"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890"} Nov 02 20:15:30.842082 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:30.841839 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerStarted","Data":"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543"} Nov 02 20:15:30.842082 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:30.841875 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerStarted","Data":"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2"} Nov 02 20:15:30.842082 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:30.841899 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerStarted","Data":"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e"} Nov 02 20:15:30.965125 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:30.965085 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:15:31.381272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:31.381240 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:31.387499 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:31.387472 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bb5949b9c-jjt6t" Nov 02 20:15:32.034805 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:32.034774 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gjf9l" Nov 02 20:15:32.619872 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:32.619843 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:32.656879 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:15:32.656845 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:15:33.349960 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:33.349927 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:33.354254 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:33.354227 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-ctj2p" Nov 02 20:15:33.789559 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:33.789487 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:37.036597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:37.036565 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:37.037085 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:37.036800 5420 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 20:15:42.251803 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:42.251714 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:42.269975 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:42.269944 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:42.901861 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:42.901832 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:15:52.214224 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:52.214195 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8plx8" Nov 02 20:15:55.898150 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:15:55.898111 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wnw5s" Nov 02 20:16:00.967860 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:00.967829 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-89w6h" Nov 02 20:16:02.656878 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:02.656839 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:16:11.742484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.742452 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:16:11.743065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.742761 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="prometheus" containerID="cri-o://ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" gracePeriod=600 Nov 02 20:16:11.743065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.742836 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy-web" containerID="cri-o://0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" gracePeriod=600 Nov 02 20:16:11.743065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.742853 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="config-reloader" containerID="cri-o://74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" gracePeriod=600 Nov 02 20:16:11.743065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.742861 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy" containerID="cri-o://ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" gracePeriod=600 Nov 02 20:16:11.743065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.743017 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="thanos-sidecar" containerID="cri-o://94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" gracePeriod=600 Nov 02 20:16:11.743366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.743258 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" gracePeriod=600 Nov 02 20:16:11.974191 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.974163 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:11.986026 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986001 5420 generic.go:356] "Generic (PLEG): container finished" podID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" exitCode=0 Nov 02 20:16:11.986026 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986025 5420 generic.go:356] "Generic (PLEG): container finished" podID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" exitCode=0 Nov 02 20:16:11.986186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986036 5420 generic.go:356] "Generic (PLEG): container finished" podID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" exitCode=0 Nov 02 20:16:11.986186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986045 5420 generic.go:356] "Generic (PLEG): container finished" podID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" exitCode=0 Nov 02 20:16:11.986186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986051 5420 generic.go:356] "Generic (PLEG): container finished" podID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" exitCode=0 Nov 02 20:16:11.986186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986058 5420 generic.go:356] "Generic (PLEG): container finished" podID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" exitCode=0 Nov 02 20:16:11.986186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986069 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:11.986186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986078 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerDied","Data":"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b"} Nov 02 20:16:11.986186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986152 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerDied","Data":"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d"} Nov 02 20:16:11.986186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986180 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerDied","Data":"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890"} Nov 02 20:16:11.986557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986207 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerDied","Data":"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543"} Nov 02 20:16:11.986557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986235 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerDied","Data":"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2"} Nov 02 20:16:11.986557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986238 5420 scope.go:122] "RemoveContainer" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" Nov 02 20:16:11.986557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986259 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerDied","Data":"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e"} Nov 02 20:16:11.986557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.986289 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"21dbeb5e-76a3-4e56-9843-60aaeb743e39","Type":"ContainerDied","Data":"6b4b83c9cc89dc8a01bf188266c0fe664964175b508c0adf9b9092e97ff001f0"} Nov 02 20:16:11.995200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:11.995186 5420 scope.go:122] "RemoveContainer" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" Nov 02 20:16:12.001636 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.001621 5420 scope.go:122] "RemoveContainer" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" Nov 02 20:16:12.007871 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.007851 5420 scope.go:122] "RemoveContainer" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" Nov 02 20:16:12.017283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.017266 5420 scope.go:122] "RemoveContainer" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" Nov 02 20:16:12.023503 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.023487 5420 scope.go:122] "RemoveContainer" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" Nov 02 20:16:12.030059 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.030041 5420 scope.go:122] "RemoveContainer" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" Nov 02 20:16:12.034028 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034010 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034037 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034068 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034107 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034145 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034188 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034220 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034261 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034289 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034320 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034345 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034371 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034405 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034437 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034475 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034482 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:12.034853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034508 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034539 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034569 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034600 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034646 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets\") pod \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\" (UID: \"21dbeb5e-76a3-4e56-9843-60aaeb743e39\") " Nov 02 20:16:12.034853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034817 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:12.035434 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034918 5420 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-metrics-client-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.035434 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.034934 5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.037225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.035666 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:12.037225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.036525 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:12.037225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.036943 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.037225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.037051 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:12.037225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.037211 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.037518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.037357 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:12.037518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.037448 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:12.037518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.037512 5420 scope.go:122] "RemoveContainer" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" Nov 02 20:16:12.037900 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:12.037864 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": container with ID starting with a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b not found: ID does not exist" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" Nov 02 20:16:12.037987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.037909 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b"} err="failed to get container status \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": rpc error: code = NotFound desc = could not find container \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": container with ID starting with a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b not found: ID does not exist" Nov 02 20:16:12.037987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.037948 5420 scope.go:122] "RemoveContainer" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" Nov 02 20:16:12.038113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.038079 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:16:12.038302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.038202 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:12.038302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.038275 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.038407 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.038329 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "kube-api-access-jn5dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:16:12.040251 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:12.039993 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": container with ID starting with ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d not found: ID does not exist" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" Nov 02 20:16:12.040251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040032 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d"} err="failed to get container status \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": rpc error: code = NotFound desc = could not find container \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": container with ID starting with ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d not found: ID does not exist" Nov 02 20:16:12.040251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040058 5420 scope.go:122] "RemoveContainer" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" Nov 02 20:16:12.040443 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:12.040297 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": container with ID starting with 0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890 not found: ID does not exist" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" Nov 02 20:16:12.040443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040320 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890"} err="failed to get container status \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": rpc error: code = NotFound desc = could not find container \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": container with ID starting with 0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890 not found: ID does not exist" Nov 02 20:16:12.040443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040337 5420 scope.go:122] "RemoveContainer" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" Nov 02 20:16:12.040443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040337 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.040443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040368 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.040690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040466 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.040690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040544 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.040690 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:12.040627 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": container with ID starting with 94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543 not found: ID does not exist" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" Nov 02 20:16:12.040690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040656 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543"} err="failed to get container status \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": rpc error: code = NotFound desc = could not find container \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": container with ID starting with 94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543 not found: ID does not exist" Nov 02 20:16:12.040690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040687 5420 scope.go:122] "RemoveContainer" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" Nov 02 20:16:12.040924 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040700 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.040924 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040708 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:16:12.040987 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:12.040923 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": container with ID starting with 74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2 not found: ID does not exist" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" Nov 02 20:16:12.040987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040949 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2"} err="failed to get container status \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": rpc error: code = NotFound desc = could not find container \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": container with ID starting with 74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2 not found: ID does not exist" Nov 02 20:16:12.040987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.040963 5420 scope.go:122] "RemoveContainer" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" Nov 02 20:16:12.041206 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:12.041190 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": container with ID starting with ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e not found: ID does not exist" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" Nov 02 20:16:12.041270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.041214 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e"} err="failed to get container status \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": rpc error: code = NotFound desc = could not find container \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": container with ID starting with ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e not found: ID does not exist" Nov 02 20:16:12.041270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.041235 5420 scope.go:122] "RemoveContainer" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" Nov 02 20:16:12.041472 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:12.041456 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": container with ID starting with 2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda not found: ID does not exist" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" Nov 02 20:16:12.041516 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.041475 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda"} err="failed to get container status \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": rpc error: code = NotFound desc = could not find container \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": container with ID starting with 2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda not found: ID does not exist" Nov 02 20:16:12.041516 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.041488 5420 scope.go:122] "RemoveContainer" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" Nov 02 20:16:12.041876 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.041850 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b"} err="failed to get container status \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": rpc error: code = NotFound desc = could not find container \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": container with ID starting with a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b not found: ID does not exist" Nov 02 20:16:12.041955 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.041877 5420 scope.go:122] "RemoveContainer" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" Nov 02 20:16:12.042104 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042087 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d"} err="failed to get container status \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": rpc error: code = NotFound desc = could not find container \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": container with ID starting with ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d not found: ID does not exist" Nov 02 20:16:12.042166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042104 5420 scope.go:122] "RemoveContainer" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" Nov 02 20:16:12.042350 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042333 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890"} err="failed to get container status \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": rpc error: code = NotFound desc = could not find container \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": container with ID starting with 0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890 not found: ID does not exist" Nov 02 20:16:12.042350 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042349 5420 scope.go:122] "RemoveContainer" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" Nov 02 20:16:12.042576 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042558 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543"} err="failed to get container status \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": rpc error: code = NotFound desc = could not find container \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": container with ID starting with 94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543 not found: ID does not exist" Nov 02 20:16:12.042642 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042578 5420 scope.go:122] "RemoveContainer" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" Nov 02 20:16:12.042804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042783 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2"} err="failed to get container status \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": rpc error: code = NotFound desc = could not find container \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": container with ID starting with 74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2 not found: ID does not exist" Nov 02 20:16:12.042865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042805 5420 scope.go:122] "RemoveContainer" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" Nov 02 20:16:12.043011 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.042989 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e"} err="failed to get container status \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": rpc error: code = NotFound desc = could not find container \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": container with ID starting with ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e not found: ID does not exist" Nov 02 20:16:12.043079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.043012 5420 scope.go:122] "RemoveContainer" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" Nov 02 20:16:12.043221 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.043204 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda"} err="failed to get container status \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": rpc error: code = NotFound desc = could not find container \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": container with ID starting with 2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda not found: ID does not exist" Nov 02 20:16:12.043286 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.043221 5420 scope.go:122] "RemoveContainer" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" Nov 02 20:16:12.043453 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.043433 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b"} err="failed to get container status \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": rpc error: code = NotFound desc = could not find container \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": container with ID starting with a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b not found: ID does not exist" Nov 02 20:16:12.043453 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.043451 5420 scope.go:122] "RemoveContainer" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" Nov 02 20:16:12.043646 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.043628 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d"} err="failed to get container status \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": rpc error: code = NotFound desc = could not find container \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": container with ID starting with ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d not found: ID does not exist" Nov 02 20:16:12.043646 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.043645 5420 scope.go:122] "RemoveContainer" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" Nov 02 20:16:12.044006 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.043990 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890"} err="failed to get container status \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": rpc error: code = NotFound desc = could not find container \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": container with ID starting with 0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890 not found: ID does not exist" Nov 02 20:16:12.044006 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044006 5420 scope.go:122] "RemoveContainer" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" Nov 02 20:16:12.044238 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044216 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543"} err="failed to get container status \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": rpc error: code = NotFound desc = could not find container \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": container with ID starting with 94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543 not found: ID does not exist" Nov 02 20:16:12.044238 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044237 5420 scope.go:122] "RemoveContainer" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" Nov 02 20:16:12.044477 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044453 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2"} err="failed to get container status \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": rpc error: code = NotFound desc = could not find container \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": container with ID starting with 74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2 not found: ID does not exist" Nov 02 20:16:12.044535 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044479 5420 scope.go:122] "RemoveContainer" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" Nov 02 20:16:12.044740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044721 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e"} err="failed to get container status \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": rpc error: code = NotFound desc = could not find container \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": container with ID starting with ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e not found: ID does not exist" Nov 02 20:16:12.044807 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044741 5420 scope.go:122] "RemoveContainer" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" Nov 02 20:16:12.044947 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044931 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda"} err="failed to get container status \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": rpc error: code = NotFound desc = could not find container \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": container with ID starting with 2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda not found: ID does not exist" Nov 02 20:16:12.044999 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.044946 5420 scope.go:122] "RemoveContainer" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" Nov 02 20:16:12.045165 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.045139 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b"} err="failed to get container status \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": rpc error: code = NotFound desc = could not find container \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": container with ID starting with a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b not found: ID does not exist" Nov 02 20:16:12.045165 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.045164 5420 scope.go:122] "RemoveContainer" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" Nov 02 20:16:12.045362 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.045345 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d"} err="failed to get container status \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": rpc error: code = NotFound desc = could not find container \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": container with ID starting with ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d not found: ID does not exist" Nov 02 20:16:12.045362 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.045361 5420 scope.go:122] "RemoveContainer" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" Nov 02 20:16:12.045563 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.045547 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890"} err="failed to get container status \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": rpc error: code = NotFound desc = could not find container \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": container with ID starting with 0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890 not found: ID does not exist" Nov 02 20:16:12.045563 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.045563 5420 scope.go:122] "RemoveContainer" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" Nov 02 20:16:12.045857 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.045833 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543"} err="failed to get container status \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": rpc error: code = NotFound desc = could not find container \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": container with ID starting with 94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543 not found: ID does not exist" Nov 02 20:16:12.045919 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.045860 5420 scope.go:122] "RemoveContainer" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" Nov 02 20:16:12.046095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.046075 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2"} err="failed to get container status \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": rpc error: code = NotFound desc = could not find container \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": container with ID starting with 74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2 not found: ID does not exist" Nov 02 20:16:12.046142 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.046097 5420 scope.go:122] "RemoveContainer" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" Nov 02 20:16:12.046416 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.046393 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e"} err="failed to get container status \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": rpc error: code = NotFound desc = could not find container \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": container with ID starting with ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e not found: ID does not exist" Nov 02 20:16:12.046487 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.046418 5420 scope.go:122] "RemoveContainer" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" Nov 02 20:16:12.046697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.046658 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda"} err="failed to get container status \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": rpc error: code = NotFound desc = could not find container \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": container with ID starting with 2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda not found: ID does not exist" Nov 02 20:16:12.046747 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.046697 5420 scope.go:122] "RemoveContainer" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" Nov 02 20:16:12.046922 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.046905 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b"} err="failed to get container status \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": rpc error: code = NotFound desc = could not find container \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": container with ID starting with a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b not found: ID does not exist" Nov 02 20:16:12.046996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.046924 5420 scope.go:122] "RemoveContainer" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" Nov 02 20:16:12.047151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047134 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d"} err="failed to get container status \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": rpc error: code = NotFound desc = could not find container \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": container with ID starting with ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d not found: ID does not exist" Nov 02 20:16:12.047192 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047152 5420 scope.go:122] "RemoveContainer" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" Nov 02 20:16:12.047377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047361 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890"} err="failed to get container status \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": rpc error: code = NotFound desc = could not find container \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": container with ID starting with 0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890 not found: ID does not exist" Nov 02 20:16:12.047438 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047378 5420 scope.go:122] "RemoveContainer" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" Nov 02 20:16:12.047627 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047613 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543"} err="failed to get container status \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": rpc error: code = NotFound desc = could not find container \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": container with ID starting with 94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543 not found: ID does not exist" Nov 02 20:16:12.047697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047628 5420 scope.go:122] "RemoveContainer" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" Nov 02 20:16:12.047821 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047805 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2"} err="failed to get container status \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": rpc error: code = NotFound desc = could not find container \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": container with ID starting with 74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2 not found: ID does not exist" Nov 02 20:16:12.047866 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047821 5420 scope.go:122] "RemoveContainer" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" Nov 02 20:16:12.048001 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.047986 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e"} err="failed to get container status \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": rpc error: code = NotFound desc = could not find container \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": container with ID starting with ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e not found: ID does not exist" Nov 02 20:16:12.048038 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048002 5420 scope.go:122] "RemoveContainer" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" Nov 02 20:16:12.048188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048170 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda"} err="failed to get container status \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": rpc error: code = NotFound desc = could not find container \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": container with ID starting with 2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda not found: ID does not exist" Nov 02 20:16:12.048254 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048189 5420 scope.go:122] "RemoveContainer" containerID="a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b" Nov 02 20:16:12.048382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048367 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b"} err="failed to get container status \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": rpc error: code = NotFound desc = could not find container \"a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b\": container with ID starting with a8a2f7ae6ee8004aaa92d1b9a227accea45ad2570a1ecfbeb5dac03a92d3457b not found: ID does not exist" Nov 02 20:16:12.048425 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048381 5420 scope.go:122] "RemoveContainer" containerID="ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d" Nov 02 20:16:12.048606 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048582 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d"} err="failed to get container status \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": rpc error: code = NotFound desc = could not find container \"ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d\": container with ID starting with ba54b92241a11fb4d6e1fe426ccbbcce320491230c5d07cf18f9729d7085d00d not found: ID does not exist" Nov 02 20:16:12.048649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048608 5420 scope.go:122] "RemoveContainer" containerID="0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890" Nov 02 20:16:12.048859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048842 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890"} err="failed to get container status \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": rpc error: code = NotFound desc = could not find container \"0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890\": container with ID starting with 0cb9fc2a39ab31700beeb7d495ccf4fa3009e220a7c7f2bc845283f23dbef890 not found: ID does not exist" Nov 02 20:16:12.048902 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.048860 5420 scope.go:122] "RemoveContainer" containerID="94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543" Nov 02 20:16:12.049150 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.049129 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543"} err="failed to get container status \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": rpc error: code = NotFound desc = could not find container \"94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543\": container with ID starting with 94a6104c83e7fdba77602e841dbccb04c6cba9587894169184ed822d71b64543 not found: ID does not exist" Nov 02 20:16:12.049219 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.049151 5420 scope.go:122] "RemoveContainer" containerID="74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2" Nov 02 20:16:12.049377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.049360 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2"} err="failed to get container status \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": rpc error: code = NotFound desc = could not find container \"74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2\": container with ID starting with 74bab6462cecccebd8d7f9f4654e9fe9da4e68cf678faca0ed76c579f3d767c2 not found: ID does not exist" Nov 02 20:16:12.049440 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.049376 5420 scope.go:122] "RemoveContainer" containerID="ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e" Nov 02 20:16:12.049584 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.049570 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e"} err="failed to get container status \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": rpc error: code = NotFound desc = could not find container \"ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e\": container with ID starting with ea9534e75890c438c1f3fe3c09a94650bf8bd4a3909f8e3685ddefc0c63d966e not found: ID does not exist" Nov 02 20:16:12.049584 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.049584 5420 scope.go:122] "RemoveContainer" containerID="2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda" Nov 02 20:16:12.049807 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.049792 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda"} err="failed to get container status \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": rpc error: code = NotFound desc = could not find container \"2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda\": container with ID starting with 2fe100d6a15c6fbcfa740f00b90684c0a488feb938dd1bd014a8b5931d465fda not found: ID does not exist" Nov 02 20:16:12.050536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.050521 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config" pod "21dbeb5e-76a3-4e56-9843-60aaeb743e39" (UID: "21dbeb5e-76a3-4e56-9843-60aaeb743e39"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:16:12.135641 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135611 5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-db\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135641 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135642 5420 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config-out\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135697 5420 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-web-config\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135707 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn5dr\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-kube-api-access-jn5dr\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135717 5420 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-config\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135726 5420 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-kube-rbac-proxy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135734 5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135744 5420 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135753 5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135762 5420 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135770 5420 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135780 5420 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-grpc-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135789 5420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/21dbeb5e-76a3-4e56-9843-60aaeb743e39-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135797 5420 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-metrics-client-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135805 5420 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21dbeb5e-76a3-4e56-9843-60aaeb743e39-tls-assets\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135815 5420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135825 5420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.135836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.135834 5420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/21dbeb5e-76a3-4e56-9843-60aaeb743e39-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:16:12.399747 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.399663 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:16:12.403593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.403570 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:16:12.436873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.436850 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:16:12.437135 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437123 5420 cpu_manager.go:414] "containerMap: removing container" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy-web" Nov 02 20:16:12.437183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437136 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy-web" Nov 02 20:16:12.437183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437151 5420 cpu_manager.go:414] "containerMap: removing container" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy-thanos" Nov 02 20:16:12.437183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437157 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy-thanos" Nov 02 20:16:12.437183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437165 5420 cpu_manager.go:414] "containerMap: removing container" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="init-config-reloader" Nov 02 20:16:12.437183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437170 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="init-config-reloader" Nov 02 20:16:12.437183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437182 5420 cpu_manager.go:414] "containerMap: removing container" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="thanos-sidecar" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437186 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="thanos-sidecar" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437194 5420 cpu_manager.go:414] "containerMap: removing container" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="config-reloader" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437198 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="config-reloader" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437208 5420 cpu_manager.go:414] "containerMap: removing container" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="prometheus" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437213 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="prometheus" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437220 5420 cpu_manager.go:414] "containerMap: removing container" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437225 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437274 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="thanos-sidecar" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437282 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="config-reloader" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437288 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="prometheus" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437293 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437300 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy-web" Nov 02 20:16:12.437382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.437305 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" containerName="kube-rbac-proxy-thanos" Nov 02 20:16:12.441141 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.441126 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.490091 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490066 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 20:16:12.490233 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490171 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 20:16:12.490294 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490275 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 20:16:12.490350 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490310 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 20:16:12.490809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490786 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 20:16:12.490809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490805 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 20:16:12.490938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490808 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 20:16:12.490938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490924 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 20:16:12.491017 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490951 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 20:16:12.491017 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490951 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 20:16:12.491017 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.490955 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dfwqd\"" Nov 02 20:16:12.491120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.491052 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 20:16:12.491161 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.491118 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 20:16:12.491216 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.491202 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aa2ub3sv54fn3\"" Nov 02 20:16:12.491270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.491260 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 20:16:12.537043 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537012 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 20:16:12.537151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537123 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 20:16:12.537197 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537161 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537197 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537190 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-config\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537205 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537219 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537287 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537394 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537371 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537433 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537487 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537472 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537500 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537561 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537657 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537806 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537722 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537806 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537771 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537811 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537863 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmp4p\" (UniqueName: \"kubernetes.io/projected/0c193ef7-627a-4721-988c-2c72e7473b1c-kube-api-access-hmp4p\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537877 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:16:12.537985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537911 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.537985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.537945 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.538079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.538016 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c193ef7-627a-4721-988c-2c72e7473b1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.538079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.538047 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c193ef7-627a-4721-988c-2c72e7473b1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.538174 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.538092 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639128 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639100 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639131 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639183 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c193ef7-627a-4721-988c-2c72e7473b1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639202 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c193ef7-627a-4721-988c-2c72e7473b1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639224 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639243 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639263 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-config\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639287 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639566 5420 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 20:16:12.639733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639575 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639749 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639823 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.639960 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639939 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.639980 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640021 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640069 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640097 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640121 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640158 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640203 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640256 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640279 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.640478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.640303 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmp4p\" (UniqueName: \"kubernetes.io/projected/0c193ef7-627a-4721-988c-2c72e7473b1c-kube-api-access-hmp4p\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.641137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.641110 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.642154 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.642123 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.642244 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.642175 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.642311 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.642290 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.642526 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.642507 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0c193ef7-627a-4721-988c-2c72e7473b1c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.643484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.643389 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.643616 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.643506 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.643967 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.643842 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0c193ef7-627a-4721-988c-2c72e7473b1c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.643967 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.643910 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.644261 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.644222 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.644261 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.644231 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.644411 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.644347 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0c193ef7-627a-4721-988c-2c72e7473b1c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.644497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.644479 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-config\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.644691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.644657 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.644751 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.644715 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.644751 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.644720 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.645090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.645074 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0c193ef7-627a-4721-988c-2c72e7473b1c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.697162 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.697134 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmp4p\" (UniqueName: \"kubernetes.io/projected/0c193ef7-627a-4721-988c-2c72e7473b1c-kube-api-access-hmp4p\") pod \"prometheus-k8s-0\" (UID: \"0c193ef7-627a-4721-988c-2c72e7473b1c\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:12.764598 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:12.764540 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:13.183117 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:13.183084 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 20:16:13.186748 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:16:13.186720 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c193ef7_627a_4721_988c_2c72e7473b1c.slice/crio-f848c71eb66baed7e96b7ad3e1d551e75f59eddcdcfbf1bacdb7b930e4b6121e WatchSource:0}: Error finding container f848c71eb66baed7e96b7ad3e1d551e75f59eddcdcfbf1bacdb7b930e4b6121e: Status 404 returned error can't find the container with id f848c71eb66baed7e96b7ad3e1d551e75f59eddcdcfbf1bacdb7b930e4b6121e Nov 02 20:16:13.775366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:13.775334 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21dbeb5e-76a3-4e56-9843-60aaeb743e39" path="/var/lib/kubelet/pods/21dbeb5e-76a3-4e56-9843-60aaeb743e39/volumes" Nov 02 20:16:13.994565 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:13.994528 5420 generic.go:356] "Generic (PLEG): container finished" podID="0c193ef7-627a-4721-988c-2c72e7473b1c" containerID="ad327f414bcca9596b7eb36e19fe49c52ea38696ce7a5184696714cb2351b8ee" exitCode=0 Nov 02 20:16:13.994768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:13.994595 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c193ef7-627a-4721-988c-2c72e7473b1c","Type":"ContainerDied","Data":"ad327f414bcca9596b7eb36e19fe49c52ea38696ce7a5184696714cb2351b8ee"} Nov 02 20:16:13.994768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:13.994651 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c193ef7-627a-4721-988c-2c72e7473b1c","Type":"ContainerStarted","Data":"f848c71eb66baed7e96b7ad3e1d551e75f59eddcdcfbf1bacdb7b930e4b6121e"} Nov 02 20:16:15.001681 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:15.001644 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c193ef7-627a-4721-988c-2c72e7473b1c","Type":"ContainerStarted","Data":"4a21362571e869505e9d000ace4a4614549f700489c8b51be6af04f4aa91a657"} Nov 02 20:16:15.002037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:15.001728 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c193ef7-627a-4721-988c-2c72e7473b1c","Type":"ContainerStarted","Data":"97a1818a3e3f8b39f652e7590c3ba4aa1672622e8b4b5156838e77aedc7dbf10"} Nov 02 20:16:15.002037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:15.001754 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c193ef7-627a-4721-988c-2c72e7473b1c","Type":"ContainerStarted","Data":"b0494c29c0a257d4b905ad5bb2fe76737c92586e21d2c6b96f834a77652b6e31"} Nov 02 20:16:15.002037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:15.001783 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c193ef7-627a-4721-988c-2c72e7473b1c","Type":"ContainerStarted","Data":"c4b04f72fbb9090dfa4cbc5fd2a0681c1ac0201b9d3c462e93465fc3e9092a0d"} Nov 02 20:16:15.002037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:15.001806 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c193ef7-627a-4721-988c-2c72e7473b1c","Type":"ContainerStarted","Data":"7da4478b294422ceeb3f3c2200e7a1238a78773950c558123dc57b759d9f0749"} Nov 02 20:16:15.002037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:15.001828 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0c193ef7-627a-4721-988c-2c72e7473b1c","Type":"ContainerStarted","Data":"bed8b44e0685a1a99acda416d2113a9609ac00fcba98d4110789d8cfaaf0c2cc"} Nov 02 20:16:15.126144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:15.126086 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.110340436 podStartE2EDuration="3.126070263s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.729827ms" isStatefulPod=false podCreationTimestamp="2026-11-02 20:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:16:15.125291231 +0000 UTC m=+47.489902437" watchObservedRunningTime="2026-11-02 20:16:15.126070263 +0000 UTC m=+47.490681466" Nov 02 20:16:17.765077 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:17.765029 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:32.655953 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:16:32.655904 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:16:55.495539 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:55.495500 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:55.510427 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:55.510402 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:16:56.150125 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:16:56.150096 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 20:17:02.656199 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:17:02.656159 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:17:32.656464 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:17:32.656385 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:18:02.656284 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:18:02.656246 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:18:32.656264 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:18:32.656228 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:19:02.654818 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:19:02.654740 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:19:28.483006 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.482975 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8"] Nov 02 20:19:28.485441 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.485426 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.580927 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.580901 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-pdntb\"" Nov 02 20:19:28.580927 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.580913 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 20:19:28.581120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.581021 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 20:19:28.581848 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.581828 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8"] Nov 02 20:19:28.582250 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.582234 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg"] Nov 02 20:19:28.584486 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.584470 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.614368 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.614345 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.614459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.614380 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfvth\" (UniqueName: \"kubernetes.io/projected/87df9f37-de0f-4a66-8af0-2f55e2f55e37-kube-api-access-gfvth\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.614459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.614399 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.633134 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.633117 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-dsqpp\"" Nov 02 20:19:28.638430 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.638414 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg"] Nov 02 20:19:28.684289 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.684270 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp"] Nov 02 20:19:28.686726 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.686711 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.715363 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.715345 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfvth\" (UniqueName: \"kubernetes.io/projected/87df9f37-de0f-4a66-8af0-2f55e2f55e37-kube-api-access-gfvth\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.715458 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.715374 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.715458 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.715405 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.715458 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.715422 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbj8f\" (UniqueName: \"kubernetes.io/projected/5044aeb1-df21-4ae7-9fb3-25702a42632f-kube-api-access-rbj8f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.715582 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.715498 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.715582 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.715533 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.715713 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.715696 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.715797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.715784 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.736179 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.736135 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rhhns\"" Nov 02 20:19:28.743569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.743546 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp"] Nov 02 20:19:28.774629 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.774607 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfvth\" (UniqueName: \"kubernetes.io/projected/87df9f37-de0f-4a66-8af0-2f55e2f55e37-kube-api-access-gfvth\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.786515 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.786495 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv"] Nov 02 20:19:28.789106 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.789091 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:28.789639 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.789621 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z"] Nov 02 20:19:28.792081 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.792065 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:28.797979 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.797961 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:19:28.816214 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.816193 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dbsl\" (UniqueName: \"kubernetes.io/projected/563f54fd-7163-4847-bcef-a7a9d917b7e6-kube-api-access-2dbsl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.816302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.816230 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.816302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.816260 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.816414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.816314 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbj8f\" (UniqueName: \"kubernetes.io/projected/5044aeb1-df21-4ae7-9fb3-25702a42632f-kube-api-access-rbj8f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.816469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.816437 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.816523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.816500 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.816634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.816612 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.816710 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.816662 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.837957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.837936 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-fzltd\"" Nov 02 20:19:28.840681 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.840539 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-jwxwl\"" Nov 02 20:19:28.847587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.846592 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv"] Nov 02 20:19:28.848730 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.848666 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z"] Nov 02 20:19:28.872995 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.872968 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbj8f\" (UniqueName: \"kubernetes.io/projected/5044aeb1-df21-4ae7-9fb3-25702a42632f-kube-api-access-rbj8f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.896779 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.896757 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:19:28.917098 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917078 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dbsl\" (UniqueName: \"kubernetes.io/projected/563f54fd-7163-4847-bcef-a7a9d917b7e6-kube-api-access-2dbsl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.917188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917108 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2zl\" (UniqueName: \"kubernetes.io/projected/d5c4ad73-b377-4fa3-87d0-bfb000487b46-kube-api-access-xj2zl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:28.917188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917139 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:28.917317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917293 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.917376 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917356 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4ps\" (UniqueName: \"kubernetes.io/projected/37f2fb28-8e67-4604-b203-4f702c610fe0-kube-api-access-xp4ps\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:28.917412 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917400 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:28.917521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917506 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.917561 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917535 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:28.917561 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917537 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.917627 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917569 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:28.917730 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.917717 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.974018 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.973996 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dbsl\" (UniqueName: \"kubernetes.io/projected/563f54fd-7163-4847-bcef-a7a9d917b7e6-kube-api-access-2dbsl\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:28.998389 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:28.998348 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:19:29.019011 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.018989 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2zl\" (UniqueName: \"kubernetes.io/projected/d5c4ad73-b377-4fa3-87d0-bfb000487b46-kube-api-access-xj2zl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:29.019107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019021 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:29.019107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019054 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4ps\" (UniqueName: \"kubernetes.io/projected/37f2fb28-8e67-4604-b203-4f702c610fe0-kube-api-access-xp4ps\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:29.019107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019075 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:29.019273 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019111 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:29.019273 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019132 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:29.019429 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019407 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:29.019465 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019447 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:29.019500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019476 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:29.019592 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.019570 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:29.077959 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.077929 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2zl\" (UniqueName: \"kubernetes.io/projected/d5c4ad73-b377-4fa3-87d0-bfb000487b46-kube-api-access-xj2zl\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:29.078044 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.078009 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4ps\" (UniqueName: \"kubernetes.io/projected/37f2fb28-8e67-4604-b203-4f702c610fe0-kube-api-access-xp4ps\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:29.103255 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.103233 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:19:29.108415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.108397 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:19:29.179719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.179629 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8"] Nov 02 20:19:29.184523 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:19:29.184479 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87df9f37_de0f_4a66_8af0_2f55e2f55e37.slice/crio-4921c0022dd3ed8c41b3cc71bdc92fc3a357554e10e426efdb11bba97dc60b1f WatchSource:0}: Error finding container 4921c0022dd3ed8c41b3cc71bdc92fc3a357554e10e426efdb11bba97dc60b1f: Status 404 returned error can't find the container with id 4921c0022dd3ed8c41b3cc71bdc92fc3a357554e10e426efdb11bba97dc60b1f Nov 02 20:19:29.259367 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.259314 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg"] Nov 02 20:19:29.262263 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:19:29.262241 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5044aeb1_df21_4ae7_9fb3_25702a42632f.slice/crio-0a7a689b8200b52228a2920901695cdda02425e9967b8e8e0a90d981d4f1122b WatchSource:0}: Error finding container 0a7a689b8200b52228a2920901695cdda02425e9967b8e8e0a90d981d4f1122b: Status 404 returned error can't find the container with id 0a7a689b8200b52228a2920901695cdda02425e9967b8e8e0a90d981d4f1122b Nov 02 20:19:29.359687 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.359603 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp"] Nov 02 20:19:29.362760 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:19:29.362739 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563f54fd_7163_4847_bcef_a7a9d917b7e6.slice/crio-a431b619e524a1dca10aab36e5174342ff165970cdcb396bb0be320d030295ad WatchSource:0}: Error finding container a431b619e524a1dca10aab36e5174342ff165970cdcb396bb0be320d030295ad: Status 404 returned error can't find the container with id a431b619e524a1dca10aab36e5174342ff165970cdcb396bb0be320d030295ad Nov 02 20:19:29.467276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.467216 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv"] Nov 02 20:19:29.470398 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.470373 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z"] Nov 02 20:19:29.470578 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:19:29.470535 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f2fb28_8e67_4604_b203_4f702c610fe0.slice/crio-e12b7096e17d85dc21f4fd5e984cef47899a0247b1aaf4177d9e7b97dbfd19d4 WatchSource:0}: Error finding container e12b7096e17d85dc21f4fd5e984cef47899a0247b1aaf4177d9e7b97dbfd19d4: Status 404 returned error can't find the container with id e12b7096e17d85dc21f4fd5e984cef47899a0247b1aaf4177d9e7b97dbfd19d4 Nov 02 20:19:29.474222 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:19:29.474203 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c4ad73_b377_4fa3_87d0_bfb000487b46.slice/crio-7dfe218ff5571a798da8a35ca13507481d0179bf14eb4c6195a40f945044ba85 WatchSource:0}: Error finding container 7dfe218ff5571a798da8a35ca13507481d0179bf14eb4c6195a40f945044ba85: Status 404 returned error can't find the container with id 7dfe218ff5571a798da8a35ca13507481d0179bf14eb4c6195a40f945044ba85 Nov 02 20:19:29.630036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.630006 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" event={"ID":"37f2fb28-8e67-4604-b203-4f702c610fe0","Type":"ContainerStarted","Data":"e12b7096e17d85dc21f4fd5e984cef47899a0247b1aaf4177d9e7b97dbfd19d4"} Nov 02 20:19:29.631069 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.631045 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" event={"ID":"563f54fd-7163-4847-bcef-a7a9d917b7e6","Type":"ContainerStarted","Data":"a431b619e524a1dca10aab36e5174342ff165970cdcb396bb0be320d030295ad"} Nov 02 20:19:29.632034 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.632015 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" event={"ID":"5044aeb1-df21-4ae7-9fb3-25702a42632f","Type":"ContainerStarted","Data":"0a7a689b8200b52228a2920901695cdda02425e9967b8e8e0a90d981d4f1122b"} Nov 02 20:19:29.633035 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.633015 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" event={"ID":"87df9f37-de0f-4a66-8af0-2f55e2f55e37","Type":"ContainerStarted","Data":"4921c0022dd3ed8c41b3cc71bdc92fc3a357554e10e426efdb11bba97dc60b1f"} Nov 02 20:19:29.634055 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:29.634035 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" event={"ID":"d5c4ad73-b377-4fa3-87d0-bfb000487b46","Type":"ContainerStarted","Data":"7dfe218ff5571a798da8a35ca13507481d0179bf14eb4c6195a40f945044ba85"} Nov 02 20:19:30.914447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:30.914409 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f"] Nov 02 20:19:30.918108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:30.918086 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.014699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.013494 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-g2hvg\"" Nov 02 20:19:31.015012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.014920 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f"] Nov 02 20:19:31.038421 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.038391 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjlsk\" (UniqueName: \"kubernetes.io/projected/7ac85f46-3138-415b-a071-edab2333ef03-kube-api-access-xjlsk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.038552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.038446 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.038552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.038474 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.139564 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.139536 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.139564 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.139567 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.139812 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.139658 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjlsk\" (UniqueName: \"kubernetes.io/projected/7ac85f46-3138-415b-a071-edab2333ef03-kube-api-access-xjlsk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.140276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.140253 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.140461 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.140447 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.208395 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.208328 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjlsk\" (UniqueName: \"kubernetes.io/projected/7ac85f46-3138-415b-a071-edab2333ef03-kube-api-access-xjlsk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.236331 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.236116 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:19:31.635650 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.634971 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f"] Nov 02 20:19:31.651984 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:31.651955 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" event={"ID":"7ac85f46-3138-415b-a071-edab2333ef03","Type":"ContainerStarted","Data":"9e0b035718c58a02939ca95e3cb9c557ef826b0ac224c63426c6a34545582614"} Nov 02 20:19:32.684929 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:19:32.684874 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:19:32.938271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:32.938197 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68"] Nov 02 20:19:32.941372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:32.941351 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.037786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.037754 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68"] Nov 02 20:19:33.037962 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.037877 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-n7rbf\"" Nov 02 20:19:33.057498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.057460 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgr7b\" (UniqueName: \"kubernetes.io/projected/397c8ecb-4643-4b89-b7a4-3ee25f692658-kube-api-access-wgr7b\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.057684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.057519 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.057684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.057595 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.159518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.158313 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.159518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.158379 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.159518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.158496 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgr7b\" (UniqueName: \"kubernetes.io/projected/397c8ecb-4643-4b89-b7a4-3ee25f692658-kube-api-access-wgr7b\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.159518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.159166 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.159518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.159462 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.226434 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.226363 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgr7b\" (UniqueName: \"kubernetes.io/projected/397c8ecb-4643-4b89-b7a4-3ee25f692658-kube-api-access-wgr7b\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.259017 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.258983 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:19:33.661264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:33.661227 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68"] Nov 02 20:19:33.666783 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:19:33.666750 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397c8ecb_4643_4b89_b7a4_3ee25f692658.slice/crio-1c14a8970a768ea7a2698affc9b77f6b49517c213861fd4bccf6e5aa7bbe8c32 WatchSource:0}: Error finding container 1c14a8970a768ea7a2698affc9b77f6b49517c213861fd4bccf6e5aa7bbe8c32: Status 404 returned error can't find the container with id 1c14a8970a768ea7a2698affc9b77f6b49517c213861fd4bccf6e5aa7bbe8c32 Nov 02 20:19:34.678721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:34.678616 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" event={"ID":"397c8ecb-4643-4b89-b7a4-3ee25f692658","Type":"ContainerStarted","Data":"1c14a8970a768ea7a2698affc9b77f6b49517c213861fd4bccf6e5aa7bbe8c32"} Nov 02 20:19:36.082107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.082062 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm"] Nov 02 20:19:36.088644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.088036 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.137496 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.136751 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg9qz\"" Nov 02 20:19:36.158533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.158500 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 20:19:36.158703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.158576 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 20:19:36.184268 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.184240 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 20:19:36.186233 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186201 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eccd3c31-d213-4763-b79b-ee970b2c334d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.186339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186250 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5nk\" (UniqueName: \"kubernetes.io/projected/eccd3c31-d213-4763-b79b-ee970b2c334d-kube-api-access-dg5nk\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.186339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186294 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.186339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186326 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.186498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186348 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.186498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186365 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm"] Nov 02 20:19:36.186498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186409 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.186633 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186554 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.186633 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186596 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.186755 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.186637 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.193198 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.193177 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r"] Nov 02 20:19:36.196166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.196148 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.258128 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.258095 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t"] Nov 02 20:19:36.261382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.261360 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.287420 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.287368 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.287863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.287403 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.287863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.287682 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.287863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.287730 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.287863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.287767 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfwd\" (UniqueName: \"kubernetes.io/projected/1114ad08-6ea5-4485-a054-9a964bae7a4e-kube-api-access-sbfwd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.287863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.287774 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.287863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.287801 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.288211 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288033 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.288211 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288129 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.291042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288366 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.291042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288404 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.291042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288430 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.291042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288455 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.291042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288495 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eccd3c31-d213-4763-b79b-ee970b2c334d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.291042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288528 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5nk\" (UniqueName: \"kubernetes.io/projected/eccd3c31-d213-4763-b79b-ee970b2c334d-kube-api-access-dg5nk\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.291042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288552 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2h2g\" (UniqueName: \"kubernetes.io/projected/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-kube-api-access-l2h2g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.291743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.291096 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eccd3c31-d213-4763-b79b-ee970b2c334d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.291743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.291224 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.291743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.291438 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.294141 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.293954 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r"] Nov 02 20:19:36.294257 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.288598 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.298732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.298640 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.307946 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.307893 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.350334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.350195 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eccd3c31-d213-4763-b79b-ee970b2c334d-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.350334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.350231 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5nk\" (UniqueName: \"kubernetes.io/projected/eccd3c31-d213-4763-b79b-ee970b2c334d-kube-api-access-dg5nk\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8vqkm\" (UID: \"eccd3c31-d213-4763-b79b-ee970b2c334d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.353824 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.353647 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t"] Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.394848 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2h2g\" (UniqueName: \"kubernetes.io/projected/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-kube-api-access-l2h2g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.394916 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.394960 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.394990 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfwd\" (UniqueName: \"kubernetes.io/projected/1114ad08-6ea5-4485-a054-9a964bae7a4e-kube-api-access-sbfwd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.395020 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.395054 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.395444 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.395940 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.396450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.396211 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.396889 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.396687 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.404690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.404422 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:36.453609 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.453578 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfwd\" (UniqueName: \"kubernetes.io/projected/1114ad08-6ea5-4485-a054-9a964bae7a4e-kube-api-access-sbfwd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.454231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.454201 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2h2g\" (UniqueName: \"kubernetes.io/projected/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-kube-api-access-l2h2g\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.515828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.515797 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:19:36.579286 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.579252 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:19:36.839401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.839336 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm"] Nov 02 20:19:36.842638 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:19:36.842604 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccd3c31_d213_4763_b79b_ee970b2c334d.slice/crio-282b98fe936bafc848d5a3fa0ed08ece62ed58d0f7f1bebc4f65397993d4f9b1 WatchSource:0}: Error finding container 282b98fe936bafc848d5a3fa0ed08ece62ed58d0f7f1bebc4f65397993d4f9b1: Status 404 returned error can't find the container with id 282b98fe936bafc848d5a3fa0ed08ece62ed58d0f7f1bebc4f65397993d4f9b1 Nov 02 20:19:36.929445 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:36.929365 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r"] Nov 02 20:19:37.012389 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:37.012304 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t"] Nov 02 20:19:37.711763 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:37.711661 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" event={"ID":"eccd3c31-d213-4763-b79b-ee970b2c334d","Type":"ContainerStarted","Data":"282b98fe936bafc848d5a3fa0ed08ece62ed58d0f7f1bebc4f65397993d4f9b1"} Nov 02 20:19:37.713319 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:37.713285 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" event={"ID":"1114ad08-6ea5-4485-a054-9a964bae7a4e","Type":"ContainerStarted","Data":"1520fdd0b04dd0d68b9f20e8d8326d1ef255a5aff99a73ef763a19d8b22d3a08"} Nov 02 20:19:37.715321 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:37.715296 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" event={"ID":"3b9e8869-5c76-45cd-90ba-ebee9bc2d243","Type":"ContainerStarted","Data":"f368a31ed14afefddfeb856875753241c6365139889fd2afdfc943998b2a307a"} Nov 02 20:19:45.800641 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.800607 5420 generic.go:356] "Generic (PLEG): container finished" podID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerID="a415b9f5f11b68bee26bf6bd9efe523b1cbf7fd814cb6b8257be5cd3fbb7459c" exitCode=0 Nov 02 20:19:45.801144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.800706 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" event={"ID":"37f2fb28-8e67-4604-b203-4f702c610fe0","Type":"ContainerDied","Data":"a415b9f5f11b68bee26bf6bd9efe523b1cbf7fd814cb6b8257be5cd3fbb7459c"} Nov 02 20:19:45.804939 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.804891 5420 generic.go:356] "Generic (PLEG): container finished" podID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerID="327a722a37674b519e77d1ada8cf7d858fa7f14aa2b10b9a3701967a770e8a22" exitCode=0 Nov 02 20:19:45.805273 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.805245 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" event={"ID":"563f54fd-7163-4847-bcef-a7a9d917b7e6","Type":"ContainerDied","Data":"327a722a37674b519e77d1ada8cf7d858fa7f14aa2b10b9a3701967a770e8a22"} Nov 02 20:19:45.807647 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.807485 5420 generic.go:356] "Generic (PLEG): container finished" podID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerID="c63db51a6e5146045d4532f343853bd778958ced9f8794de24a3574af4234cae" exitCode=0 Nov 02 20:19:45.807647 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.807571 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" event={"ID":"397c8ecb-4643-4b89-b7a4-3ee25f692658","Type":"ContainerDied","Data":"c63db51a6e5146045d4532f343853bd778958ced9f8794de24a3574af4234cae"} Nov 02 20:19:45.809934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.809484 5420 generic.go:356] "Generic (PLEG): container finished" podID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerID="3a82b3831c01d2fa02a48e7068a49e7ac73f8b9928e3886275e0f849b957cade" exitCode=0 Nov 02 20:19:45.809934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.809556 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" event={"ID":"5044aeb1-df21-4ae7-9fb3-25702a42632f","Type":"ContainerDied","Data":"3a82b3831c01d2fa02a48e7068a49e7ac73f8b9928e3886275e0f849b957cade"} Nov 02 20:19:45.814603 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.812550 5420 generic.go:356] "Generic (PLEG): container finished" podID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerID="e8e52fccf9b2ba0e7fcb5f7bc172ac02494b7fbac0bc533f1a7e6d3841edeab9" exitCode=0 Nov 02 20:19:45.814603 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.812957 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" event={"ID":"87df9f37-de0f-4a66-8af0-2f55e2f55e37","Type":"ContainerDied","Data":"e8e52fccf9b2ba0e7fcb5f7bc172ac02494b7fbac0bc533f1a7e6d3841edeab9"} Nov 02 20:19:45.820088 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.818403 5420 generic.go:356] "Generic (PLEG): container finished" podID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerID="499ab471ed368a7032085190da046c4444c18f7e2fa218d131a9ebca6065de79" exitCode=0 Nov 02 20:19:45.820088 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.818433 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" event={"ID":"d5c4ad73-b377-4fa3-87d0-bfb000487b46","Type":"ContainerDied","Data":"499ab471ed368a7032085190da046c4444c18f7e2fa218d131a9ebca6065de79"} Nov 02 20:19:45.823478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.823449 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:45.823565 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.823514 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:45.823565 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:45.823544 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:19:46.823514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:46.823487 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" event={"ID":"eccd3c31-d213-4763-b79b-ee970b2c334d","Type":"ContainerStarted","Data":"f05f20118cc98775f57607ea40da9ac9920c44b93e81416f6e0544fbc0d8b65d"} Nov 02 20:19:46.824884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:46.824861 5420 generic.go:356] "Generic (PLEG): container finished" podID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerID="7ac72984ecbdf2579370562ef100a06c7cb15696d49fddbd69af51ff2b70d346" exitCode=0 Nov 02 20:19:46.824975 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:46.824907 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" event={"ID":"3b9e8869-5c76-45cd-90ba-ebee9bc2d243","Type":"ContainerDied","Data":"7ac72984ecbdf2579370562ef100a06c7cb15696d49fddbd69af51ff2b70d346"} Nov 02 20:19:46.937810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:46.937752 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" podStartSLOduration=1.961068166 podStartE2EDuration="10.937733517s" totalImagesPullingTime="8.976665351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:19:46.935369921 +0000 UTC m=+259.299981121" watchObservedRunningTime="2026-11-02 20:19:46.937733517 +0000 UTC m=+259.302344722" Nov 02 20:19:47.405021 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.404987 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:47.409847 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.409821 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:47.833375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.833307 5420 generic.go:356] "Generic (PLEG): container finished" podID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerID="7de0d926bb7c68208a7552875e707e89152c18b6b3f6243465eb96f4a7411ebb" exitCode=0 Nov 02 20:19:47.833811 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.833397 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" event={"ID":"5044aeb1-df21-4ae7-9fb3-25702a42632f","Type":"ContainerDied","Data":"7de0d926bb7c68208a7552875e707e89152c18b6b3f6243465eb96f4a7411ebb"} Nov 02 20:19:47.837255 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.837226 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" event={"ID":"87df9f37-de0f-4a66-8af0-2f55e2f55e37","Type":"ContainerStarted","Data":"cd5a5e40a9e1ef7305680161542352289d127e14c8e4283223fdf222ef9d9a23"} Nov 02 20:19:47.841378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.841237 5420 generic.go:356] "Generic (PLEG): container finished" podID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerID="7bc77fdef477bf9b7585614b4fe0d47970ac4456ff7edeacebdcb2c7a19f4d65" exitCode=0 Nov 02 20:19:47.841378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.841332 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" event={"ID":"1114ad08-6ea5-4485-a054-9a964bae7a4e","Type":"ContainerDied","Data":"7bc77fdef477bf9b7585614b4fe0d47970ac4456ff7edeacebdcb2c7a19f4d65"} Nov 02 20:19:47.845361 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.845337 5420 generic.go:356] "Generic (PLEG): container finished" podID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerID="802c2b59688d815e387ee25b4656e2da57c967db480ca479c15d8acc9db45558" exitCode=0 Nov 02 20:19:47.845468 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.845417 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" event={"ID":"37f2fb28-8e67-4604-b203-4f702c610fe0","Type":"ContainerDied","Data":"802c2b59688d815e387ee25b4656e2da57c967db480ca479c15d8acc9db45558"} Nov 02 20:19:47.848322 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.848238 5420 generic.go:356] "Generic (PLEG): container finished" podID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerID="3731b8f1a521a71f3f790bec5e3c5062cfd9d48b0711909a499c9c6ec1437366" exitCode=0 Nov 02 20:19:47.848322 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.848271 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" event={"ID":"563f54fd-7163-4847-bcef-a7a9d917b7e6","Type":"ContainerDied","Data":"3731b8f1a521a71f3f790bec5e3c5062cfd9d48b0711909a499c9c6ec1437366"} Nov 02 20:19:47.851695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.851404 5420 generic.go:356] "Generic (PLEG): container finished" podID="7ac85f46-3138-415b-a071-edab2333ef03" containerID="42b7a9dbbfd9f1439eefec8b79b985a535dd30c820772a8b8013b1fff2d373de" exitCode=0 Nov 02 20:19:47.852829 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.852728 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" event={"ID":"7ac85f46-3138-415b-a071-edab2333ef03","Type":"ContainerDied","Data":"42b7a9dbbfd9f1439eefec8b79b985a535dd30c820772a8b8013b1fff2d373de"} Nov 02 20:19:47.852829 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.852773 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:47.853549 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:47.853529 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:19:48.863646 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:48.863609 5420 generic.go:356] "Generic (PLEG): container finished" podID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerID="cd5a5e40a9e1ef7305680161542352289d127e14c8e4283223fdf222ef9d9a23" exitCode=0 Nov 02 20:19:48.864115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:48.863801 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" event={"ID":"87df9f37-de0f-4a66-8af0-2f55e2f55e37","Type":"ContainerDied","Data":"cd5a5e40a9e1ef7305680161542352289d127e14c8e4283223fdf222ef9d9a23"} Nov 02 20:19:48.867796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:48.867772 5420 generic.go:356] "Generic (PLEG): container finished" podID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerID="09c7d91fb558ef9c230a7eb5a85761f9d6491c05d03943db61a51165952603f1" exitCode=0 Nov 02 20:19:48.867896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:48.867850 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" event={"ID":"1114ad08-6ea5-4485-a054-9a964bae7a4e","Type":"ContainerDied","Data":"09c7d91fb558ef9c230a7eb5a85761f9d6491c05d03943db61a51165952603f1"} Nov 02 20:19:48.872494 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:48.872478 5420 generic.go:356] "Generic (PLEG): container finished" podID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerID="39de53556a5adb027a62aee8b34aeb0fae8cadd9a2a03cb0f5e5b6359cab0dcd" exitCode=0 Nov 02 20:19:48.873376 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:48.873355 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" event={"ID":"3b9e8869-5c76-45cd-90ba-ebee9bc2d243","Type":"ContainerDied","Data":"39de53556a5adb027a62aee8b34aeb0fae8cadd9a2a03cb0f5e5b6359cab0dcd"} Nov 02 20:19:49.884797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:49.884302 5420 generic.go:356] "Generic (PLEG): container finished" podID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerID="587265112c1c34908d61ffefe0e48c2a8367de531bf542ea0142ec6b2cf9f7d5" exitCode=0 Nov 02 20:19:49.884797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:49.884730 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" event={"ID":"d5c4ad73-b377-4fa3-87d0-bfb000487b46","Type":"ContainerDied","Data":"587265112c1c34908d61ffefe0e48c2a8367de531bf542ea0142ec6b2cf9f7d5"} Nov 02 20:19:49.890742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:49.890413 5420 generic.go:356] "Generic (PLEG): container finished" podID="7ac85f46-3138-415b-a071-edab2333ef03" containerID="f97a5acca1424495a56337a3600d2e2ce802fe8934a720a8cb8bd22dec145b3d" exitCode=0 Nov 02 20:19:49.890742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:49.890487 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" event={"ID":"7ac85f46-3138-415b-a071-edab2333ef03","Type":"ContainerDied","Data":"f97a5acca1424495a56337a3600d2e2ce802fe8934a720a8cb8bd22dec145b3d"} Nov 02 20:19:49.896822 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:49.895867 5420 generic.go:356] "Generic (PLEG): container finished" podID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerID="463caaee1a967af2aabf2da4c5184712bf41d0203bb837f32914c5e1b77b20e0" exitCode=0 Nov 02 20:19:49.896822 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:19:49.896766 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" event={"ID":"397c8ecb-4643-4b89-b7a4-3ee25f692658","Type":"ContainerDied","Data":"463caaee1a967af2aabf2da4c5184712bf41d0203bb837f32914c5e1b77b20e0"} Nov 02 20:20:02.673176 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:20:02.672949 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:20:06.039790 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:06.039711 5420 generic.go:356] "Generic (PLEG): container finished" podID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerID="9a00d5758c089cdcac0beadbae6004108b6c02a2d492a171ea6e3d896bcb5d1a" exitCode=0 Nov 02 20:20:06.040292 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:06.039803 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" event={"ID":"37f2fb28-8e67-4604-b203-4f702c610fe0","Type":"ContainerDied","Data":"9a00d5758c089cdcac0beadbae6004108b6c02a2d492a171ea6e3d896bcb5d1a"} Nov 02 20:20:06.044931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:06.044615 5420 generic.go:356] "Generic (PLEG): container finished" podID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerID="8038e14740dbb40cd1bb3bf96ced6080eddc5def28238c04144cb750be6efdc4" exitCode=0 Nov 02 20:20:06.044931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:06.044753 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" event={"ID":"563f54fd-7163-4847-bcef-a7a9d917b7e6","Type":"ContainerDied","Data":"8038e14740dbb40cd1bb3bf96ced6080eddc5def28238c04144cb750be6efdc4"} Nov 02 20:20:06.049900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:06.049872 5420 generic.go:356] "Generic (PLEG): container finished" podID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerID="34c8e2765f0ec2f2fa346e98481b2cc4ddeb4275e48a13d8622e3526c37dbd3c" exitCode=0 Nov 02 20:20:06.050002 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:06.049975 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" event={"ID":"5044aeb1-df21-4ae7-9fb3-25702a42632f","Type":"ContainerDied","Data":"34c8e2765f0ec2f2fa346e98481b2cc4ddeb4275e48a13d8622e3526c37dbd3c"} Nov 02 20:20:07.062872 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.062809 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" event={"ID":"87df9f37-de0f-4a66-8af0-2f55e2f55e37","Type":"ContainerStarted","Data":"767986827d8e0550e861574d029198c9050cb166fafddbdf36c7fa49c36e56a0"} Nov 02 20:20:07.246749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.246461 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:20:07.358550 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.358450 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" podStartSLOduration=3.524214609 podStartE2EDuration="39.358432578s" totalImagesPullingTime="35.834217969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:19:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:07.173879974 +0000 UTC m=+279.538491179" watchObservedRunningTime="2026-11-02 20:20:07.358432578 +0000 UTC m=+279.723043785" Nov 02 20:20:07.364864 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.364834 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:20:07.370891 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.370800 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:20:07.427414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.426877 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f2fb28-8e67-4604-b203-4f702c610fe0-kube-api-access-xp4ps\" (UniqueName: \"kubernetes.io/projected/37f2fb28-8e67-4604-b203-4f702c610fe0-kube-api-access-xp4ps\") pod \"37f2fb28-8e67-4604-b203-4f702c610fe0\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " Nov 02 20:20:07.427414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.426921 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-util\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-util\") pod \"37f2fb28-8e67-4604-b203-4f702c610fe0\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " Nov 02 20:20:07.427414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.426960 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-bundle\") pod \"37f2fb28-8e67-4604-b203-4f702c610fe0\" (UID: \"37f2fb28-8e67-4604-b203-4f702c610fe0\") " Nov 02 20:20:07.428185 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.428128 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-bundle" pod "37f2fb28-8e67-4604-b203-4f702c610fe0" (UID: "37f2fb28-8e67-4604-b203-4f702c610fe0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:07.429941 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.429905 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f2fb28-8e67-4604-b203-4f702c610fe0-kube-api-access-xp4ps" pod "37f2fb28-8e67-4604-b203-4f702c610fe0" (UID: "37f2fb28-8e67-4604-b203-4f702c610fe0"). InnerVolumeSpecName "kube-api-access-xp4ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:07.432783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.432743 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-util" pod "37f2fb28-8e67-4604-b203-4f702c610fe0" (UID: "37f2fb28-8e67-4604-b203-4f702c610fe0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:07.527964 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.527936 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-bundle\") pod \"5044aeb1-df21-4ae7-9fb3-25702a42632f\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " Nov 02 20:20:07.528077 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.528003 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-bundle\") pod \"563f54fd-7163-4847-bcef-a7a9d917b7e6\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " Nov 02 20:20:07.528077 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.528045 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5044aeb1-df21-4ae7-9fb3-25702a42632f-kube-api-access-rbj8f\" (UniqueName: \"kubernetes.io/projected/5044aeb1-df21-4ae7-9fb3-25702a42632f-kube-api-access-rbj8f\") pod \"5044aeb1-df21-4ae7-9fb3-25702a42632f\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " Nov 02 20:20:07.528077 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.528072 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563f54fd-7163-4847-bcef-a7a9d917b7e6-kube-api-access-2dbsl\" (UniqueName: \"kubernetes.io/projected/563f54fd-7163-4847-bcef-a7a9d917b7e6-kube-api-access-2dbsl\") pod \"563f54fd-7163-4847-bcef-a7a9d917b7e6\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " Nov 02 20:20:07.528216 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.528112 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-util\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-util\") pod \"563f54fd-7163-4847-bcef-a7a9d917b7e6\" (UID: \"563f54fd-7163-4847-bcef-a7a9d917b7e6\") " Nov 02 20:20:07.528216 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.528161 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-util\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-util\") pod \"5044aeb1-df21-4ae7-9fb3-25702a42632f\" (UID: \"5044aeb1-df21-4ae7-9fb3-25702a42632f\") " Nov 02 20:20:07.528353 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.528329 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp4ps\" (UniqueName: \"kubernetes.io/projected/37f2fb28-8e67-4604-b203-4f702c610fe0-kube-api-access-xp4ps\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:07.528353 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.528347 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:07.528482 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.528362 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37f2fb28-8e67-4604-b203-4f702c610fe0-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:07.529082 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.529052 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-bundle" pod "5044aeb1-df21-4ae7-9fb3-25702a42632f" (UID: "5044aeb1-df21-4ae7-9fb3-25702a42632f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:07.529300 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.529275 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-bundle" pod "563f54fd-7163-4847-bcef-a7a9d917b7e6" (UID: "563f54fd-7163-4847-bcef-a7a9d917b7e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:07.532769 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.532744 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5044aeb1-df21-4ae7-9fb3-25702a42632f-kube-api-access-rbj8f" pod "5044aeb1-df21-4ae7-9fb3-25702a42632f" (UID: "5044aeb1-df21-4ae7-9fb3-25702a42632f"). InnerVolumeSpecName "kube-api-access-rbj8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:07.532869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.532842 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563f54fd-7163-4847-bcef-a7a9d917b7e6-kube-api-access-2dbsl" pod "563f54fd-7163-4847-bcef-a7a9d917b7e6" (UID: "563f54fd-7163-4847-bcef-a7a9d917b7e6"). InnerVolumeSpecName "kube-api-access-2dbsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:07.534816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.534783 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-util" pod "563f54fd-7163-4847-bcef-a7a9d917b7e6" (UID: "563f54fd-7163-4847-bcef-a7a9d917b7e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:07.534921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.534826 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-util" pod "5044aeb1-df21-4ae7-9fb3-25702a42632f" (UID: "5044aeb1-df21-4ae7-9fb3-25702a42632f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:07.629378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.629302 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:07.629378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.629331 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5044aeb1-df21-4ae7-9fb3-25702a42632f-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:07.629378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.629348 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:07.629378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.629364 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbj8f\" (UniqueName: \"kubernetes.io/projected/5044aeb1-df21-4ae7-9fb3-25702a42632f-kube-api-access-rbj8f\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:07.629378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.629380 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dbsl\" (UniqueName: \"kubernetes.io/projected/563f54fd-7163-4847-bcef-a7a9d917b7e6-kube-api-access-2dbsl\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:07.629762 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:07.629393 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/563f54fd-7163-4847-bcef-a7a9d917b7e6-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:08.068498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.068471 5420 generic.go:356] "Generic (PLEG): container finished" podID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerID="767986827d8e0550e861574d029198c9050cb166fafddbdf36c7fa49c36e56a0" exitCode=0 Nov 02 20:20:08.068833 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.068558 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" event={"ID":"87df9f37-de0f-4a66-8af0-2f55e2f55e37","Type":"ContainerDied","Data":"767986827d8e0550e861574d029198c9050cb166fafddbdf36c7fa49c36e56a0"} Nov 02 20:20:08.070615 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.070592 5420 generic.go:356] "Generic (PLEG): container finished" podID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerID="225643c590477b60ff995d5993bf203d41894c579361ead38b1162dd94c4494a" exitCode=0 Nov 02 20:20:08.070745 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.070691 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" event={"ID":"1114ad08-6ea5-4485-a054-9a964bae7a4e","Type":"ContainerDied","Data":"225643c590477b60ff995d5993bf203d41894c579361ead38b1162dd94c4494a"} Nov 02 20:20:08.072966 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.072946 5420 generic.go:356] "Generic (PLEG): container finished" podID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerID="a9b49f38ffd3c307a5d889a8385d77dabb5622b9d41ed05ea4e862632747c48e" exitCode=0 Nov 02 20:20:08.073043 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.073016 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" event={"ID":"3b9e8869-5c76-45cd-90ba-ebee9bc2d243","Type":"ContainerDied","Data":"a9b49f38ffd3c307a5d889a8385d77dabb5622b9d41ed05ea4e862632747c48e"} Nov 02 20:20:08.075080 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.075053 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" event={"ID":"37f2fb28-8e67-4604-b203-4f702c610fe0","Type":"ContainerDied","Data":"e12b7096e17d85dc21f4fd5e984cef47899a0247b1aaf4177d9e7b97dbfd19d4"} Nov 02 20:20:08.075205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.075090 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e12b7096e17d85dc21f4fd5e984cef47899a0247b1aaf4177d9e7b97dbfd19d4" Nov 02 20:20:08.075205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.075088 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czwhmv" Nov 02 20:20:08.077354 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.077336 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" Nov 02 20:20:08.077477 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.077342 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847adcctp" event={"ID":"563f54fd-7163-4847-bcef-a7a9d917b7e6","Type":"ContainerDied","Data":"a431b619e524a1dca10aab36e5174342ff165970cdcb396bb0be320d030295ad"} Nov 02 20:20:08.077579 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.077500 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a431b619e524a1dca10aab36e5174342ff165970cdcb396bb0be320d030295ad" Nov 02 20:20:08.079590 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.079569 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" event={"ID":"5044aeb1-df21-4ae7-9fb3-25702a42632f","Type":"ContainerDied","Data":"0a7a689b8200b52228a2920901695cdda02425e9967b8e8e0a90d981d4f1122b"} Nov 02 20:20:08.079692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.079606 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a7a689b8200b52228a2920901695cdda02425e9967b8e8e0a90d981d4f1122b" Nov 02 20:20:08.079768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:08.079746 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74k4dg" Nov 02 20:20:09.085608 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.085581 5420 generic.go:356] "Generic (PLEG): container finished" podID="7ac85f46-3138-415b-a071-edab2333ef03" containerID="cd0f1124031c851ff8529cd29bc4cdbfb2c64fcee6b7531590993ffb0f245e5d" exitCode=0 Nov 02 20:20:09.086004 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.085656 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" event={"ID":"7ac85f46-3138-415b-a071-edab2333ef03","Type":"ContainerDied","Data":"cd0f1124031c851ff8529cd29bc4cdbfb2c64fcee6b7531590993ffb0f245e5d"} Nov 02 20:20:09.087620 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.087602 5420 generic.go:356] "Generic (PLEG): container finished" podID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerID="c7cd331a2edf470510f33e5097741fda36828541f3656621b89e2119c9f02bad" exitCode=0 Nov 02 20:20:09.087726 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.087692 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" event={"ID":"397c8ecb-4643-4b89-b7a4-3ee25f692658","Type":"ContainerDied","Data":"c7cd331a2edf470510f33e5097741fda36828541f3656621b89e2119c9f02bad"} Nov 02 20:20:09.089627 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.089606 5420 generic.go:356] "Generic (PLEG): container finished" podID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerID="d6a6dbc8f872af5b6b6136e9720eef74c61d58bc4afd12327524fd271be87dfe" exitCode=0 Nov 02 20:20:09.089739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.089693 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" event={"ID":"d5c4ad73-b377-4fa3-87d0-bfb000487b46","Type":"ContainerDied","Data":"d6a6dbc8f872af5b6b6136e9720eef74c61d58bc4afd12327524fd271be87dfe"} Nov 02 20:20:09.249334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.249312 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:20:09.253355 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.253338 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:20:09.269511 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.269491 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:20:09.343587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.343536 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87df9f37-de0f-4a66-8af0-2f55e2f55e37-kube-api-access-gfvth\" (UniqueName: \"kubernetes.io/projected/87df9f37-de0f-4a66-8af0-2f55e2f55e37-kube-api-access-gfvth\") pod \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " Nov 02 20:20:09.343743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.343652 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-util\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-util\") pod \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " Nov 02 20:20:09.343743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.343721 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-bundle\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-bundle\") pod \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\" (UID: \"87df9f37-de0f-4a66-8af0-2f55e2f55e37\") " Nov 02 20:20:09.344134 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.344113 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-bundle" pod "87df9f37-de0f-4a66-8af0-2f55e2f55e37" (UID: "87df9f37-de0f-4a66-8af0-2f55e2f55e37"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:09.345456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.345435 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87df9f37-de0f-4a66-8af0-2f55e2f55e37-kube-api-access-gfvth" pod "87df9f37-de0f-4a66-8af0-2f55e2f55e37" (UID: "87df9f37-de0f-4a66-8af0-2f55e2f55e37"). InnerVolumeSpecName "kube-api-access-gfvth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:09.348928 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.348910 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-util" pod "87df9f37-de0f-4a66-8af0-2f55e2f55e37" (UID: "87df9f37-de0f-4a66-8af0-2f55e2f55e37"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:09.444348 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444327 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-util\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-util\") pod \"1114ad08-6ea5-4485-a054-9a964bae7a4e\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " Nov 02 20:20:09.444438 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444361 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-bundle\") pod \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " Nov 02 20:20:09.444438 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444393 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-kube-api-access-l2h2g\" (UniqueName: \"kubernetes.io/projected/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-kube-api-access-l2h2g\") pod \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " Nov 02 20:20:09.444438 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444412 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1114ad08-6ea5-4485-a054-9a964bae7a4e-kube-api-access-sbfwd\" (UniqueName: \"kubernetes.io/projected/1114ad08-6ea5-4485-a054-9a964bae7a4e-kube-api-access-sbfwd\") pod \"1114ad08-6ea5-4485-a054-9a964bae7a4e\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " Nov 02 20:20:09.444551 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444438 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-bundle\") pod \"1114ad08-6ea5-4485-a054-9a964bae7a4e\" (UID: \"1114ad08-6ea5-4485-a054-9a964bae7a4e\") " Nov 02 20:20:09.444551 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444464 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-util\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-util\") pod \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\" (UID: \"3b9e8869-5c76-45cd-90ba-ebee9bc2d243\") " Nov 02 20:20:09.444691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444643 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.444770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444696 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87df9f37-de0f-4a66-8af0-2f55e2f55e37-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.444770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444738 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfvth\" (UniqueName: \"kubernetes.io/projected/87df9f37-de0f-4a66-8af0-2f55e2f55e37-kube-api-access-gfvth\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.444873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.444857 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-bundle" pod "3b9e8869-5c76-45cd-90ba-ebee9bc2d243" (UID: "3b9e8869-5c76-45cd-90ba-ebee9bc2d243"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:09.445101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.445072 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-bundle" pod "1114ad08-6ea5-4485-a054-9a964bae7a4e" (UID: "1114ad08-6ea5-4485-a054-9a964bae7a4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:09.446352 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.446328 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-kube-api-access-l2h2g" pod "3b9e8869-5c76-45cd-90ba-ebee9bc2d243" (UID: "3b9e8869-5c76-45cd-90ba-ebee9bc2d243"). InnerVolumeSpecName "kube-api-access-l2h2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:09.446437 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.446379 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1114ad08-6ea5-4485-a054-9a964bae7a4e-kube-api-access-sbfwd" pod "1114ad08-6ea5-4485-a054-9a964bae7a4e" (UID: "1114ad08-6ea5-4485-a054-9a964bae7a4e"). InnerVolumeSpecName "kube-api-access-sbfwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:09.449207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.449183 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-util" pod "3b9e8869-5c76-45cd-90ba-ebee9bc2d243" (UID: "3b9e8869-5c76-45cd-90ba-ebee9bc2d243"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:09.450132 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.450114 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-util" pod "1114ad08-6ea5-4485-a054-9a964bae7a4e" (UID: "1114ad08-6ea5-4485-a054-9a964bae7a4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:09.545391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.545371 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.545391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.545391 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2h2g\" (UniqueName: \"kubernetes.io/projected/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-kube-api-access-l2h2g\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.545512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.545400 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbfwd\" (UniqueName: \"kubernetes.io/projected/1114ad08-6ea5-4485-a054-9a964bae7a4e-kube-api-access-sbfwd\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.545512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.545409 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.545512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.545417 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b9e8869-5c76-45cd-90ba-ebee9bc2d243-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:09.545512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:09.545424 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1114ad08-6ea5-4485-a054-9a964bae7a4e-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.094639 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.094616 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" Nov 02 20:20:10.094985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.094656 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6sq5s8" event={"ID":"87df9f37-de0f-4a66-8af0-2f55e2f55e37","Type":"ContainerDied","Data":"4921c0022dd3ed8c41b3cc71bdc92fc3a357554e10e426efdb11bba97dc60b1f"} Nov 02 20:20:10.094985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.094707 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4921c0022dd3ed8c41b3cc71bdc92fc3a357554e10e426efdb11bba97dc60b1f" Nov 02 20:20:10.096686 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.096652 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" event={"ID":"1114ad08-6ea5-4485-a054-9a964bae7a4e","Type":"ContainerDied","Data":"1520fdd0b04dd0d68b9f20e8d8326d1ef255a5aff99a73ef763a19d8b22d3a08"} Nov 02 20:20:10.096774 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.096704 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1520fdd0b04dd0d68b9f20e8d8326d1ef255a5aff99a73ef763a19d8b22d3a08" Nov 02 20:20:10.096774 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.096763 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cvk25r" Nov 02 20:20:10.098560 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.098545 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" Nov 02 20:20:10.098658 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.098581 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6tst4t" event={"ID":"3b9e8869-5c76-45cd-90ba-ebee9bc2d243","Type":"ContainerDied","Data":"f368a31ed14afefddfeb856875753241c6365139889fd2afdfc943998b2a307a"} Nov 02 20:20:10.098658 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.098611 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f368a31ed14afefddfeb856875753241c6365139889fd2afdfc943998b2a307a" Nov 02 20:20:10.247851 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.247824 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:20:10.283769 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.283737 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:20:10.289885 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.289864 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:20:10.350375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.350327 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-util\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-util\") pod \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " Nov 02 20:20:10.350375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.350359 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c4ad73-b377-4fa3-87d0-bfb000487b46-kube-api-access-xj2zl\" (UniqueName: \"kubernetes.io/projected/d5c4ad73-b377-4fa3-87d0-bfb000487b46-kube-api-access-xj2zl\") pod \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " Nov 02 20:20:10.350500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.350389 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-bundle\") pod \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\" (UID: \"d5c4ad73-b377-4fa3-87d0-bfb000487b46\") " Nov 02 20:20:10.351100 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.351076 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-bundle" pod "d5c4ad73-b377-4fa3-87d0-bfb000487b46" (UID: "d5c4ad73-b377-4fa3-87d0-bfb000487b46"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:10.352213 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.352193 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c4ad73-b377-4fa3-87d0-bfb000487b46-kube-api-access-xj2zl" pod "d5c4ad73-b377-4fa3-87d0-bfb000487b46" (UID: "d5c4ad73-b377-4fa3-87d0-bfb000487b46"). InnerVolumeSpecName "kube-api-access-xj2zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:10.355566 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.355463 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-util" pod "d5c4ad73-b377-4fa3-87d0-bfb000487b46" (UID: "d5c4ad73-b377-4fa3-87d0-bfb000487b46"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:10.450754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.450731 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-bundle\") pod \"7ac85f46-3138-415b-a071-edab2333ef03\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " Nov 02 20:20:10.450873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.450784 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-util\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-util\") pod \"397c8ecb-4643-4b89-b7a4-3ee25f692658\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " Nov 02 20:20:10.450873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.450811 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac85f46-3138-415b-a071-edab2333ef03-kube-api-access-xjlsk\" (UniqueName: \"kubernetes.io/projected/7ac85f46-3138-415b-a071-edab2333ef03-kube-api-access-xjlsk\") pod \"7ac85f46-3138-415b-a071-edab2333ef03\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " Nov 02 20:20:10.450873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.450845 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-bundle\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-bundle\") pod \"397c8ecb-4643-4b89-b7a4-3ee25f692658\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " Nov 02 20:20:10.450873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.450864 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-util\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-util\") pod \"7ac85f46-3138-415b-a071-edab2333ef03\" (UID: \"7ac85f46-3138-415b-a071-edab2333ef03\") " Nov 02 20:20:10.451078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.450890 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397c8ecb-4643-4b89-b7a4-3ee25f692658-kube-api-access-wgr7b\" (UniqueName: \"kubernetes.io/projected/397c8ecb-4643-4b89-b7a4-3ee25f692658-kube-api-access-wgr7b\") pod \"397c8ecb-4643-4b89-b7a4-3ee25f692658\" (UID: \"397c8ecb-4643-4b89-b7a4-3ee25f692658\") " Nov 02 20:20:10.451127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.451098 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.451127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.451118 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2zl\" (UniqueName: \"kubernetes.io/projected/d5c4ad73-b377-4fa3-87d0-bfb000487b46-kube-api-access-xj2zl\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.451228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.451132 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5c4ad73-b377-4fa3-87d0-bfb000487b46-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.451277 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.451257 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-bundle" pod "397c8ecb-4643-4b89-b7a4-3ee25f692658" (UID: "397c8ecb-4643-4b89-b7a4-3ee25f692658"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:10.451446 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.451413 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-bundle" pod "7ac85f46-3138-415b-a071-edab2333ef03" (UID: "7ac85f46-3138-415b-a071-edab2333ef03"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:10.452921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.452900 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac85f46-3138-415b-a071-edab2333ef03-kube-api-access-xjlsk" pod "7ac85f46-3138-415b-a071-edab2333ef03" (UID: "7ac85f46-3138-415b-a071-edab2333ef03"). InnerVolumeSpecName "kube-api-access-xjlsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:10.452982 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.452930 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397c8ecb-4643-4b89-b7a4-3ee25f692658-kube-api-access-wgr7b" pod "397c8ecb-4643-4b89-b7a4-3ee25f692658" (UID: "397c8ecb-4643-4b89-b7a4-3ee25f692658"). InnerVolumeSpecName "kube-api-access-wgr7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:20:10.459127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.459099 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-util" pod "7ac85f46-3138-415b-a071-edab2333ef03" (UID: "7ac85f46-3138-415b-a071-edab2333ef03"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:10.459201 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.459140 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-util" pod "397c8ecb-4643-4b89-b7a4-3ee25f692658" (UID: "397c8ecb-4643-4b89-b7a4-3ee25f692658"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:20:10.551952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.551933 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.552029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.551953 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjlsk\" (UniqueName: \"kubernetes.io/projected/7ac85f46-3138-415b-a071-edab2333ef03-kube-api-access-xjlsk\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.552029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.551964 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/397c8ecb-4643-4b89-b7a4-3ee25f692658-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.552029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.551971 5420 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-util\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.552029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.551979 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgr7b\" (UniqueName: \"kubernetes.io/projected/397c8ecb-4643-4b89-b7a4-3ee25f692658-kube-api-access-wgr7b\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:10.552029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:10.551986 5420 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ac85f46-3138-415b-a071-edab2333ef03-bundle\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:20:11.105518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.105497 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" Nov 02 20:20:11.105906 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.105534 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458t68" event={"ID":"397c8ecb-4643-4b89-b7a4-3ee25f692658","Type":"ContainerDied","Data":"1c14a8970a768ea7a2698affc9b77f6b49517c213861fd4bccf6e5aa7bbe8c32"} Nov 02 20:20:11.105906 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.105584 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c14a8970a768ea7a2698affc9b77f6b49517c213861fd4bccf6e5aa7bbe8c32" Nov 02 20:20:11.107520 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.107501 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" event={"ID":"d5c4ad73-b377-4fa3-87d0-bfb000487b46","Type":"ContainerDied","Data":"7dfe218ff5571a798da8a35ca13507481d0179bf14eb4c6195a40f945044ba85"} Nov 02 20:20:11.107626 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.107546 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dfe218ff5571a798da8a35ca13507481d0179bf14eb4c6195a40f945044ba85" Nov 02 20:20:11.107626 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.107508 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmwx9z" Nov 02 20:20:11.109310 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.109291 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" event={"ID":"7ac85f46-3138-415b-a071-edab2333ef03","Type":"ContainerDied","Data":"9e0b035718c58a02939ca95e3cb9c557ef826b0ac224c63426c6a34545582614"} Nov 02 20:20:11.109405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.109326 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e0b035718c58a02939ca95e3cb9c557ef826b0ac224c63426c6a34545582614" Nov 02 20:20:11.109405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:11.109347 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71pfc4f" Nov 02 20:20:15.180859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.180824 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n"] Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181105 5420 cpu_manager.go:414] "containerMap: removing container" podUID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181115 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181122 5420 cpu_manager.go:414] "containerMap: removing container" podUID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181127 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181133 5420 cpu_manager.go:414] "containerMap: removing container" podUID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181138 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181147 5420 cpu_manager.go:414] "containerMap: removing container" podUID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181152 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181158 5420 cpu_manager.go:414] "containerMap: removing container" podUID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181161 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181172 5420 cpu_manager.go:414] "containerMap: removing container" podUID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181176 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181182 5420 cpu_manager.go:414] "containerMap: removing container" podUID="7ac85f46-3138-415b-a071-edab2333ef03" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181186 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac85f46-3138-415b-a071-edab2333ef03" containerName="util" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181193 5420 cpu_manager.go:414] "containerMap: removing container" podUID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerName="pull" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181197 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerName="pull" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181204 5420 cpu_manager.go:414] "containerMap: removing container" podUID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181207 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181215 5420 cpu_manager.go:414] "containerMap: removing container" podUID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerName="pull" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181219 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerName="pull" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181226 5420 cpu_manager.go:414] "containerMap: removing container" podUID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerName="pull" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181230 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerName="pull" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181236 5420 cpu_manager.go:414] "containerMap: removing container" podUID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181240 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181246 5420 cpu_manager.go:414] "containerMap: removing container" podUID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181250 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181257 5420 cpu_manager.go:414] "containerMap: removing container" podUID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerName="extract" Nov 02 20:20:15.181251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181261 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181268 5420 cpu_manager.go:414] "containerMap: removing container" podUID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181272 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181281 5420 cpu_manager.go:414] "containerMap: removing container" podUID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181285 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181291 5420 cpu_manager.go:414] "containerMap: removing container" podUID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerName="util" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181295 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerName="util" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181300 5420 cpu_manager.go:414] "containerMap: removing container" podUID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181304 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181310 5420 cpu_manager.go:414] "containerMap: removing container" podUID="7ac85f46-3138-415b-a071-edab2333ef03" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181314 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac85f46-3138-415b-a071-edab2333ef03" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181319 5420 cpu_manager.go:414] "containerMap: removing container" podUID="7ac85f46-3138-415b-a071-edab2333ef03" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181323 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac85f46-3138-415b-a071-edab2333ef03" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181333 5420 cpu_manager.go:414] "containerMap: removing container" podUID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerName="util" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181338 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerName="util" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181343 5420 cpu_manager.go:414] "containerMap: removing container" podUID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181346 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181352 5420 cpu_manager.go:414] "containerMap: removing container" podUID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181356 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181361 5420 cpu_manager.go:414] "containerMap: removing container" podUID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181365 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerName="pull" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181373 5420 cpu_manager.go:414] "containerMap: removing container" podUID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181380 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181389 5420 cpu_manager.go:414] "containerMap: removing container" podUID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181396 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181406 5420 cpu_manager.go:414] "containerMap: removing container" podUID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerName="util" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181410 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerName="util" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181469 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="5044aeb1-df21-4ae7-9fb3-25702a42632f" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181477 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="397c8ecb-4643-4b89-b7a4-3ee25f692658" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181483 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9e8869-5c76-45cd-90ba-ebee9bc2d243" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181489 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1114ad08-6ea5-4485-a054-9a964bae7a4e" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181494 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c4ad73-b377-4fa3-87d0-bfb000487b46" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181499 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="87df9f37-de0f-4a66-8af0-2f55e2f55e37" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181506 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="563f54fd-7163-4847-bcef-a7a9d917b7e6" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181513 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f2fb28-8e67-4604-b203-4f702c610fe0" containerName="extract" Nov 02 20:20:15.182019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.181518 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac85f46-3138-415b-a071-edab2333ef03" containerName="extract" Nov 02 20:20:15.184637 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.184618 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" Nov 02 20:20:15.279958 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.279933 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:20:15.280110 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.280094 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 20:20:15.280163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.280143 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7s26g\"" Nov 02 20:20:15.281330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.281314 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n"] Nov 02 20:20:15.283581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.283563 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ade6eebc-e4f6-4c8e-876d-8bb2801c9a94-tmp\") pod \"jobset-operator-674cd59dc5-bp47n\" (UID: \"ade6eebc-e4f6-4c8e-876d-8bb2801c9a94\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" Nov 02 20:20:15.283636 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.283623 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92rcq\" (UniqueName: \"kubernetes.io/projected/ade6eebc-e4f6-4c8e-876d-8bb2801c9a94-kube-api-access-92rcq\") pod \"jobset-operator-674cd59dc5-bp47n\" (UID: \"ade6eebc-e4f6-4c8e-876d-8bb2801c9a94\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" Nov 02 20:20:15.384560 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.384525 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92rcq\" (UniqueName: \"kubernetes.io/projected/ade6eebc-e4f6-4c8e-876d-8bb2801c9a94-kube-api-access-92rcq\") pod \"jobset-operator-674cd59dc5-bp47n\" (UID: \"ade6eebc-e4f6-4c8e-876d-8bb2801c9a94\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" Nov 02 20:20:15.384726 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.384572 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ade6eebc-e4f6-4c8e-876d-8bb2801c9a94-tmp\") pod \"jobset-operator-674cd59dc5-bp47n\" (UID: \"ade6eebc-e4f6-4c8e-876d-8bb2801c9a94\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" Nov 02 20:20:15.385019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.384995 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ade6eebc-e4f6-4c8e-876d-8bb2801c9a94-tmp\") pod \"jobset-operator-674cd59dc5-bp47n\" (UID: \"ade6eebc-e4f6-4c8e-876d-8bb2801c9a94\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" Nov 02 20:20:15.442656 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.442592 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92rcq\" (UniqueName: \"kubernetes.io/projected/ade6eebc-e4f6-4c8e-876d-8bb2801c9a94-kube-api-access-92rcq\") pod \"jobset-operator-674cd59dc5-bp47n\" (UID: \"ade6eebc-e4f6-4c8e-876d-8bb2801c9a94\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" Nov 02 20:20:15.503103 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.503069 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" Nov 02 20:20:15.867117 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:15.866996 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n"] Nov 02 20:20:15.870030 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:20:15.870000 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podade6eebc_e4f6_4c8e_876d_8bb2801c9a94.slice/crio-c641b541c5e2c868e131b893e2f1a11de06845f107a4a79481161565ddd7ee69 WatchSource:0}: Error finding container c641b541c5e2c868e131b893e2f1a11de06845f107a4a79481161565ddd7ee69: Status 404 returned error can't find the container with id c641b541c5e2c868e131b893e2f1a11de06845f107a4a79481161565ddd7ee69 Nov 02 20:20:16.130859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:16.130775 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" event={"ID":"ade6eebc-e4f6-4c8e-876d-8bb2801c9a94","Type":"ContainerStarted","Data":"c641b541c5e2c868e131b893e2f1a11de06845f107a4a79481161565ddd7ee69"} Nov 02 20:20:18.930740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:18.929099 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56"] Nov 02 20:20:18.934133 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:18.932412 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.009069 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.009031 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-metrics-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.009069 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.009068 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.009299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.009090 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npcdp\" (UniqueName: \"kubernetes.io/projected/b25e61b2-6090-4851-98c3-70e205aab847-kube-api-access-npcdp\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.009299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.009199 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25e61b2-6090-4851-98c3-70e205aab847-tmp\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.009299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.009231 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b25e61b2-6090-4851-98c3-70e205aab847-manifests\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.009299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.009265 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-webhook-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.061481 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.061451 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 20:20:19.061481 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.061484 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 20:20:19.061714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.061636 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 20:20:19.061750 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.061712 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 20:20:19.061750 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.061716 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 20:20:19.061814 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.061769 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-72vcs\"" Nov 02 20:20:19.109853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.109823 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25e61b2-6090-4851-98c3-70e205aab847-tmp\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.110050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.109866 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b25e61b2-6090-4851-98c3-70e205aab847-manifests\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.110050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.109987 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-webhook-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.110050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.110041 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-metrics-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.110226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.110069 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.110226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.110104 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npcdp\" (UniqueName: \"kubernetes.io/projected/b25e61b2-6090-4851-98c3-70e205aab847-kube-api-access-npcdp\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.110427 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.110403 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b25e61b2-6090-4851-98c3-70e205aab847-manifests\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.112302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.112279 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b25e61b2-6090-4851-98c3-70e205aab847-tmp\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.112536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.112513 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-webhook-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.112611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.112571 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-metrics-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.112687 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.112632 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b25e61b2-6090-4851-98c3-70e205aab847-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.174630 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.174499 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56"] Nov 02 20:20:19.179197 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.179169 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npcdp\" (UniqueName: \"kubernetes.io/projected/b25e61b2-6090-4851-98c3-70e205aab847-kube-api-access-npcdp\") pod \"opendatahub-operator-controller-manager-5984b9787d-hcg56\" (UID: \"b25e61b2-6090-4851-98c3-70e205aab847\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.249057 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.248966 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:19.724550 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:19.724514 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56"] Nov 02 20:20:19.727848 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:20:19.727816 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25e61b2_6090_4851_98c3_70e205aab847.slice/crio-d959663a36a4dc563495ac1af84a45372e872458e5561de1e6b65cdb54672244 WatchSource:0}: Error finding container d959663a36a4dc563495ac1af84a45372e872458e5561de1e6b65cdb54672244: Status 404 returned error can't find the container with id d959663a36a4dc563495ac1af84a45372e872458e5561de1e6b65cdb54672244 Nov 02 20:20:20.149991 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:20.149952 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" event={"ID":"ade6eebc-e4f6-4c8e-876d-8bb2801c9a94","Type":"ContainerStarted","Data":"031a1001c368ed7ea7cc34c290588026bff5a35e8e55a03d19bc3143e53e4e84"} Nov 02 20:20:20.151194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:20.151168 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" event={"ID":"b25e61b2-6090-4851-98c3-70e205aab847","Type":"ContainerStarted","Data":"d959663a36a4dc563495ac1af84a45372e872458e5561de1e6b65cdb54672244"} Nov 02 20:20:20.258645 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:20.258543 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-bp47n" podStartSLOduration=1.175973201 podStartE2EDuration="5.258527608s" totalImagesPullingTime="4.082554407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:20.257549111 +0000 UTC m=+292.622160316" watchObservedRunningTime="2026-11-02 20:20:20.258527608 +0000 UTC m=+292.623138812" Nov 02 20:20:24.172734 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:24.172701 5420 generic.go:356] "Generic (PLEG): container finished" podID="b25e61b2-6090-4851-98c3-70e205aab847" containerID="b23824a0e670ec8d5f1ef94dcf23006e0a8481a15e123112741301d87f487691" exitCode=0 Nov 02 20:20:24.173125 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:24.172743 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" event={"ID":"b25e61b2-6090-4851-98c3-70e205aab847","Type":"ContainerDied","Data":"b23824a0e670ec8d5f1ef94dcf23006e0a8481a15e123112741301d87f487691"} Nov 02 20:20:25.178573 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:25.178532 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" event={"ID":"b25e61b2-6090-4851-98c3-70e205aab847","Type":"ContainerStarted","Data":"67f1b6d041a237ce38021bcf05f8b30c7e75e0a9a0383fa1b3c9d2929407ac1c"} Nov 02 20:20:25.179007 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:25.178784 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:25.309902 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:25.309849 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" podStartSLOduration=2.229604948 podStartE2EDuration="7.309835333s" totalImagesPullingTime="4.615542462s" totalInitContainerRuntime="464.687923ms" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:25.307840115 +0000 UTC m=+297.672451320" watchObservedRunningTime="2026-11-02 20:20:25.309835333 +0000 UTC m=+297.674446624" Nov 02 20:20:28.520512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.520474 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql"] Nov 02 20:20:28.529027 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.529000 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" Nov 02 20:20:28.586823 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.586791 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7kj\" (UniqueName: \"kubernetes.io/projected/06ff9d5a-cfc4-4115-9ed8-e595ee874bd2-kube-api-access-8h7kj\") pod \"limitador-operator-controller-manager-5778bb4449-xr5ql\" (UID: \"06ff9d5a-cfc4-4115-9ed8-e595ee874bd2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" Nov 02 20:20:28.627613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.627580 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-495d2\"" Nov 02 20:20:28.627901 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.627882 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 20:20:28.628056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.628037 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 20:20:28.629271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.629251 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql"] Nov 02 20:20:28.687439 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.687405 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7kj\" (UniqueName: \"kubernetes.io/projected/06ff9d5a-cfc4-4115-9ed8-e595ee874bd2-kube-api-access-8h7kj\") pod \"limitador-operator-controller-manager-5778bb4449-xr5ql\" (UID: \"06ff9d5a-cfc4-4115-9ed8-e595ee874bd2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" Nov 02 20:20:28.746114 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.746083 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7kj\" (UniqueName: \"kubernetes.io/projected/06ff9d5a-cfc4-4115-9ed8-e595ee874bd2-kube-api-access-8h7kj\") pod \"limitador-operator-controller-manager-5778bb4449-xr5ql\" (UID: \"06ff9d5a-cfc4-4115-9ed8-e595ee874bd2\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" Nov 02 20:20:28.840190 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:28.840164 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" Nov 02 20:20:29.216974 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:29.216951 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql"] Nov 02 20:20:29.219499 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:20:29.219467 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ff9d5a_cfc4_4115_9ed8_e595ee874bd2.slice/crio-d1e9cba35fd53551fb05baac21cc8970f9057cfaad49218295e33f98494bef09 WatchSource:0}: Error finding container d1e9cba35fd53551fb05baac21cc8970f9057cfaad49218295e33f98494bef09: Status 404 returned error can't find the container with id d1e9cba35fd53551fb05baac21cc8970f9057cfaad49218295e33f98494bef09 Nov 02 20:20:29.221875 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:29.221856 5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:20:30.206030 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:30.205990 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" event={"ID":"06ff9d5a-cfc4-4115-9ed8-e595ee874bd2","Type":"ContainerStarted","Data":"d1e9cba35fd53551fb05baac21cc8970f9057cfaad49218295e33f98494bef09"} Nov 02 20:20:32.157434 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.157402 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49"] Nov 02 20:20:32.160769 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.160753 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" Nov 02 20:20:32.210262 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.210229 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 02 20:20:32.212734 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.212708 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49"] Nov 02 20:20:32.213508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.213486 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 02 20:20:32.220405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.220380 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df7957f-536d-4960-8261-d2d96049f93f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-r2m49\" (UID: \"6df7957f-536d-4960-8261-d2d96049f93f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" Nov 02 20:20:32.220522 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.220466 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfs4\" (UniqueName: \"kubernetes.io/projected/6df7957f-536d-4960-8261-d2d96049f93f-kube-api-access-tlfs4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-r2m49\" (UID: \"6df7957f-536d-4960-8261-d2d96049f93f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" Nov 02 20:20:32.257032 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.256999 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-44btk\"" Nov 02 20:20:32.321931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.321893 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfs4\" (UniqueName: \"kubernetes.io/projected/6df7957f-536d-4960-8261-d2d96049f93f-kube-api-access-tlfs4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-r2m49\" (UID: \"6df7957f-536d-4960-8261-d2d96049f93f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" Nov 02 20:20:32.322097 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.322012 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df7957f-536d-4960-8261-d2d96049f93f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-r2m49\" (UID: \"6df7957f-536d-4960-8261-d2d96049f93f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" Nov 02 20:20:32.322320 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.322303 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df7957f-536d-4960-8261-d2d96049f93f-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-r2m49\" (UID: \"6df7957f-536d-4960-8261-d2d96049f93f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" Nov 02 20:20:32.381359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.381329 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfs4\" (UniqueName: \"kubernetes.io/projected/6df7957f-536d-4960-8261-d2d96049f93f-kube-api-access-tlfs4\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-r2m49\" (UID: \"6df7957f-536d-4960-8261-d2d96049f93f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" Nov 02 20:20:32.472076 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.472003 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" Nov 02 20:20:32.660853 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:20:32.660818 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:20:32.900182 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:32.900144 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49"] Nov 02 20:20:32.903738 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:20:32.903697 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df7957f_536d_4960_8261_d2d96049f93f.slice/crio-fc34ae30e619c6ce829cd15d98672a3ea3da14dad7cd874b9121c9bafcca65a7 WatchSource:0}: Error finding container fc34ae30e619c6ce829cd15d98672a3ea3da14dad7cd874b9121c9bafcca65a7: Status 404 returned error can't find the container with id fc34ae30e619c6ce829cd15d98672a3ea3da14dad7cd874b9121c9bafcca65a7 Nov 02 20:20:33.224186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:33.224095 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" event={"ID":"6df7957f-536d-4960-8261-d2d96049f93f","Type":"ContainerStarted","Data":"fc34ae30e619c6ce829cd15d98672a3ea3da14dad7cd874b9121c9bafcca65a7"} Nov 02 20:20:34.230152 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:34.230117 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" event={"ID":"06ff9d5a-cfc4-4115-9ed8-e595ee874bd2","Type":"ContainerStarted","Data":"80729096124d9c418b93019cc7508f112b74818f9b88a22ecc6d54d047bbb858"} Nov 02 20:20:34.230639 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:34.230260 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" Nov 02 20:20:34.340241 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:34.340171 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" podStartSLOduration=1.796909933 podStartE2EDuration="6.340156807s" totalImagesPullingTime="4.543246874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:34.339316504 +0000 UTC m=+306.703927709" watchObservedRunningTime="2026-11-02 20:20:34.340156807 +0000 UTC m=+306.704768013" Nov 02 20:20:36.185846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:36.185818 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" Nov 02 20:20:36.241524 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:36.241443 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" event={"ID":"6df7957f-536d-4960-8261-d2d96049f93f","Type":"ContainerStarted","Data":"10f236b00ce87f43a5fa852593ae6c65e97f0a0725a37b2e76822d5e6d7f22ee"} Nov 02 20:20:36.524174 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:36.524070 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-r2m49" podStartSLOduration=1.452476549 podStartE2EDuration="4.524054162s" totalImagesPullingTime="3.071577613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:36.522186847 +0000 UTC m=+308.886798051" watchObservedRunningTime="2026-11-02 20:20:36.524054162 +0000 UTC m=+308.888665370" Nov 02 20:20:38.521158 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.521122 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl"] Nov 02 20:20:38.524531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.524506 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:38.575275 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.575244 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3addeeb8-7674-42fa-afa2-0875018ec613-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zwnzl\" (UID: \"3addeeb8-7674-42fa-afa2-0875018ec613\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:38.575413 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.575304 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plb6k\" (UniqueName: \"kubernetes.io/projected/3addeeb8-7674-42fa-afa2-0875018ec613-kube-api-access-plb6k\") pod \"cert-manager-webhook-5dbbbf75cc-zwnzl\" (UID: \"3addeeb8-7674-42fa-afa2-0875018ec613\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:38.623890 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.623855 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gh298\"" Nov 02 20:20:38.624049 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.623936 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 20:20:38.624049 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.624009 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 20:20:38.625472 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.625453 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl"] Nov 02 20:20:38.675929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.675901 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plb6k\" (UniqueName: \"kubernetes.io/projected/3addeeb8-7674-42fa-afa2-0875018ec613-kube-api-access-plb6k\") pod \"cert-manager-webhook-5dbbbf75cc-zwnzl\" (UID: \"3addeeb8-7674-42fa-afa2-0875018ec613\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:38.676081 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.675967 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3addeeb8-7674-42fa-afa2-0875018ec613-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zwnzl\" (UID: \"3addeeb8-7674-42fa-afa2-0875018ec613\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:38.734305 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.734271 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3addeeb8-7674-42fa-afa2-0875018ec613-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zwnzl\" (UID: \"3addeeb8-7674-42fa-afa2-0875018ec613\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:38.734460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.734369 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plb6k\" (UniqueName: \"kubernetes.io/projected/3addeeb8-7674-42fa-afa2-0875018ec613-kube-api-access-plb6k\") pod \"cert-manager-webhook-5dbbbf75cc-zwnzl\" (UID: \"3addeeb8-7674-42fa-afa2-0875018ec613\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:38.836175 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:38.836147 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:39.203569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:39.203546 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl"] Nov 02 20:20:39.206109 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:20:39.206082 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3addeeb8_7674_42fa_afa2_0875018ec613.slice/crio-9767d4ec767ef1196bddda863a11abb782e98f02ea504aa4d652ef29778f6bc6 WatchSource:0}: Error finding container 9767d4ec767ef1196bddda863a11abb782e98f02ea504aa4d652ef29778f6bc6: Status 404 returned error can't find the container with id 9767d4ec767ef1196bddda863a11abb782e98f02ea504aa4d652ef29778f6bc6 Nov 02 20:20:39.257111 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:39.257080 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" event={"ID":"3addeeb8-7674-42fa-afa2-0875018ec613","Type":"ContainerStarted","Data":"9767d4ec767ef1196bddda863a11abb782e98f02ea504aa4d652ef29778f6bc6"} Nov 02 20:20:43.277367 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:43.277277 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" event={"ID":"3addeeb8-7674-42fa-afa2-0875018ec613","Type":"ContainerStarted","Data":"3171d2214aa9ce4cf3f55a92ae758d56a8a1850d746e17d581f9b9a5113546f7"} Nov 02 20:20:43.277836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:43.277377 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:43.386230 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:43.386182 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" podStartSLOduration=1.581919149 podStartE2EDuration="5.386167924s" totalImagesPullingTime="3.804248775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:43.38492103 +0000 UTC m=+315.749532233" watchObservedRunningTime="2026-11-02 20:20:43.386167924 +0000 UTC m=+315.750779129" Nov 02 20:20:45.237265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:45.237238 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-xr5ql" Nov 02 20:20:48.173536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.173505 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-l2rwm"] Nov 02 20:20:48.176999 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.176975 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-l2rwm" Nov 02 20:20:48.225444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.225420 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-68qgr\"" Nov 02 20:20:48.259995 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.259966 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxsmk\" (UniqueName: \"kubernetes.io/projected/183b73c5-7d81-47f2-9a4f-43dc951085af-kube-api-access-lxsmk\") pod \"cert-manager-597b95cb84-l2rwm\" (UID: \"183b73c5-7d81-47f2-9a4f-43dc951085af\") " pod="cert-manager/cert-manager-597b95cb84-l2rwm" Nov 02 20:20:48.260131 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.260019 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/183b73c5-7d81-47f2-9a4f-43dc951085af-bound-sa-token\") pod \"cert-manager-597b95cb84-l2rwm\" (UID: \"183b73c5-7d81-47f2-9a4f-43dc951085af\") " pod="cert-manager/cert-manager-597b95cb84-l2rwm" Nov 02 20:20:48.273235 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.273207 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-l2rwm"] Nov 02 20:20:48.360780 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.360752 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/183b73c5-7d81-47f2-9a4f-43dc951085af-bound-sa-token\") pod \"cert-manager-597b95cb84-l2rwm\" (UID: \"183b73c5-7d81-47f2-9a4f-43dc951085af\") " pod="cert-manager/cert-manager-597b95cb84-l2rwm" Nov 02 20:20:48.360924 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.360817 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxsmk\" (UniqueName: \"kubernetes.io/projected/183b73c5-7d81-47f2-9a4f-43dc951085af-kube-api-access-lxsmk\") pod \"cert-manager-597b95cb84-l2rwm\" (UID: \"183b73c5-7d81-47f2-9a4f-43dc951085af\") " pod="cert-manager/cert-manager-597b95cb84-l2rwm" Nov 02 20:20:48.417437 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.417409 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/183b73c5-7d81-47f2-9a4f-43dc951085af-bound-sa-token\") pod \"cert-manager-597b95cb84-l2rwm\" (UID: \"183b73c5-7d81-47f2-9a4f-43dc951085af\") " pod="cert-manager/cert-manager-597b95cb84-l2rwm" Nov 02 20:20:48.417641 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.417624 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxsmk\" (UniqueName: \"kubernetes.io/projected/183b73c5-7d81-47f2-9a4f-43dc951085af-kube-api-access-lxsmk\") pod \"cert-manager-597b95cb84-l2rwm\" (UID: \"183b73c5-7d81-47f2-9a4f-43dc951085af\") " pod="cert-manager/cert-manager-597b95cb84-l2rwm" Nov 02 20:20:48.488326 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.488245 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-l2rwm" Nov 02 20:20:48.853619 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:48.853583 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-l2rwm"] Nov 02 20:20:48.857648 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:20:48.857619 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod183b73c5_7d81_47f2_9a4f_43dc951085af.slice/crio-18f5170fc67dce3ee46cc15b67126f2c42b1c972a93288598a32fa87c3876212 WatchSource:0}: Error finding container 18f5170fc67dce3ee46cc15b67126f2c42b1c972a93288598a32fa87c3876212: Status 404 returned error can't find the container with id 18f5170fc67dce3ee46cc15b67126f2c42b1c972a93288598a32fa87c3876212 Nov 02 20:20:49.284934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:49.284851 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zwnzl" Nov 02 20:20:49.311220 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:49.311191 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-l2rwm" event={"ID":"183b73c5-7d81-47f2-9a4f-43dc951085af","Type":"ContainerStarted","Data":"f6f569156338b29e30616b3874874c931133ce5921fd6727d8610095a53504ca"} Nov 02 20:20:49.311348 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:49.311229 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-l2rwm" event={"ID":"183b73c5-7d81-47f2-9a4f-43dc951085af","Type":"ContainerStarted","Data":"18f5170fc67dce3ee46cc15b67126f2c42b1c972a93288598a32fa87c3876212"} Nov 02 20:20:49.508089 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:49.508039 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-l2rwm" podStartSLOduration=1.508024855 podStartE2EDuration="1.508024855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:49.50694042 +0000 UTC m=+321.871551618" watchObservedRunningTime="2026-11-02 20:20:49.508024855 +0000 UTC m=+321.872636058" Nov 02 20:20:53.116236 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.116196 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz"] Nov 02 20:20:53.119564 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.119545 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.201978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.201941 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49e58208-6c5c-44d3-9750-42b6884a3325-tls-certs\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.202176 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.202041 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e58208-6c5c-44d3-9750-42b6884a3325-tmp\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.202176 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.202150 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc42w\" (UniqueName: \"kubernetes.io/projected/49e58208-6c5c-44d3-9750-42b6884a3325-kube-api-access-rc42w\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.218078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.218052 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 20:20:53.218078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.218076 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 20:20:53.218243 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.218138 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4bpw7\"" Nov 02 20:20:53.219659 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.219640 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz"] Nov 02 20:20:53.303332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.303301 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49e58208-6c5c-44d3-9750-42b6884a3325-tls-certs\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.303489 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.303354 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e58208-6c5c-44d3-9750-42b6884a3325-tmp\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.303489 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.303392 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc42w\" (UniqueName: \"kubernetes.io/projected/49e58208-6c5c-44d3-9750-42b6884a3325-kube-api-access-rc42w\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.305634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.305611 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e58208-6c5c-44d3-9750-42b6884a3325-tmp\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.305779 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.305759 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49e58208-6c5c-44d3-9750-42b6884a3325-tls-certs\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.363492 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.363463 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc42w\" (UniqueName: \"kubernetes.io/projected/49e58208-6c5c-44d3-9750-42b6884a3325-kube-api-access-rc42w\") pod \"kube-auth-proxy-676d5b6d89-lrjtz\" (UID: \"49e58208-6c5c-44d3-9750-42b6884a3325\") " pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.431391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.431326 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" Nov 02 20:20:53.797101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:53.796966 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz"] Nov 02 20:20:53.800999 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:20:53.800960 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e58208_6c5c_44d3_9750_42b6884a3325.slice/crio-28f41dbdd163a8adbf8a6585613fad8b378b91801804be44991999c50963e2df WatchSource:0}: Error finding container 28f41dbdd163a8adbf8a6585613fad8b378b91801804be44991999c50963e2df: Status 404 returned error can't find the container with id 28f41dbdd163a8adbf8a6585613fad8b378b91801804be44991999c50963e2df Nov 02 20:20:54.335540 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:54.335497 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" event={"ID":"49e58208-6c5c-44d3-9750-42b6884a3325","Type":"ContainerStarted","Data":"28f41dbdd163a8adbf8a6585613fad8b378b91801804be44991999c50963e2df"} Nov 02 20:20:57.352460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:57.352414 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" event={"ID":"49e58208-6c5c-44d3-9750-42b6884a3325","Type":"ContainerStarted","Data":"4b00688255d316c14a907be31611877cbf42416e597a2eebdb0fdc822315046c"} Nov 02 20:20:57.461108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:57.461053 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-676d5b6d89-lrjtz" podStartSLOduration=1.392178425 podStartE2EDuration="4.461040605s" totalImagesPullingTime="3.06886218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:20:57.460333624 +0000 UTC m=+329.824944830" watchObservedRunningTime="2026-11-02 20:20:57.461040605 +0000 UTC m=+329.825651810" Nov 02 20:20:58.044527 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.044493 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m"] Nov 02 20:20:58.047592 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.047569 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:20:58.096250 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.096217 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-njndk\"" Nov 02 20:20:58.143755 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.143716 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m"] Nov 02 20:20:58.146245 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.146222 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snfjn\" (UniqueName: \"kubernetes.io/projected/7d490bf4-b010-4edc-b4b6-a65fc0e2087d-kube-api-access-snfjn\") pod \"kuadrant-operator-controller-manager-678d87766f-2r66m\" (UID: \"7d490bf4-b010-4edc-b4b6-a65fc0e2087d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:20:58.146387 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.146367 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d490bf4-b010-4edc-b4b6-a65fc0e2087d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2r66m\" (UID: \"7d490bf4-b010-4edc-b4b6-a65fc0e2087d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:20:58.247628 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.247595 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d490bf4-b010-4edc-b4b6-a65fc0e2087d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2r66m\" (UID: \"7d490bf4-b010-4edc-b4b6-a65fc0e2087d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:20:58.247804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.247649 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snfjn\" (UniqueName: \"kubernetes.io/projected/7d490bf4-b010-4edc-b4b6-a65fc0e2087d-kube-api-access-snfjn\") pod \"kuadrant-operator-controller-manager-678d87766f-2r66m\" (UID: \"7d490bf4-b010-4edc-b4b6-a65fc0e2087d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:20:58.248056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.248033 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7d490bf4-b010-4edc-b4b6-a65fc0e2087d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2r66m\" (UID: \"7d490bf4-b010-4edc-b4b6-a65fc0e2087d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:20:58.304643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.304609 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snfjn\" (UniqueName: \"kubernetes.io/projected/7d490bf4-b010-4edc-b4b6-a65fc0e2087d-kube-api-access-snfjn\") pod \"kuadrant-operator-controller-manager-678d87766f-2r66m\" (UID: \"7d490bf4-b010-4edc-b4b6-a65fc0e2087d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:20:58.359447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.359426 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:20:58.820714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:58.820684 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m"] Nov 02 20:20:58.823834 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:20:58.823794 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d490bf4_b010_4edc_b4b6_a65fc0e2087d.slice/crio-f89cb8375ae0994af3c6565b08e36a28c0930b24fff8addf2cd186eefa46f895 WatchSource:0}: Error finding container f89cb8375ae0994af3c6565b08e36a28c0930b24fff8addf2cd186eefa46f895: Status 404 returned error can't find the container with id f89cb8375ae0994af3c6565b08e36a28c0930b24fff8addf2cd186eefa46f895 Nov 02 20:20:59.362853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:20:59.362818 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" event={"ID":"7d490bf4-b010-4edc-b4b6-a65fc0e2087d","Type":"ContainerStarted","Data":"f89cb8375ae0994af3c6565b08e36a28c0930b24fff8addf2cd186eefa46f895"} Nov 02 20:21:02.659021 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:21:02.658976 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:21:07.403094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:07.402997 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" event={"ID":"7d490bf4-b010-4edc-b4b6-a65fc0e2087d","Type":"ContainerStarted","Data":"2633736cf9ef8f520b7e3c26c917b01fe075a9ad7c81525a080cf114eb1d7ae1"} Nov 02 20:21:07.403094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:07.403090 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:21:07.606291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:07.606235 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" podStartSLOduration=2.364011084 podStartE2EDuration="10.606222107s" totalImagesPullingTime="8.242211023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:07.559556212 +0000 UTC m=+339.924167417" watchObservedRunningTime="2026-11-02 20:21:07.606222107 +0000 UTC m=+339.970833311" Nov 02 20:21:17.889132 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.889099 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff"] Nov 02 20:21:17.891856 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.891838 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:17.908144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.908107 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf565282-2490-4e6a-af4b-d85efbf5bab3-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:17.908144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.908151 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf565282-2490-4e6a-af4b-d85efbf5bab3-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:17.908334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.908239 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcljb\" (UniqueName: \"kubernetes.io/projected/cf565282-2490-4e6a-af4b-d85efbf5bab3-kube-api-access-gcljb\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:17.908424 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.908343 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf565282-2490-4e6a-af4b-d85efbf5bab3-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:17.940471 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.940440 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 20:21:17.940608 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.940592 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 20:21:17.986839 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:17.986813 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-c27nq\"" Nov 02 20:21:18.009346 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.009318 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf565282-2490-4e6a-af4b-d85efbf5bab3-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.009456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.009356 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf565282-2490-4e6a-af4b-d85efbf5bab3-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.009456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.009388 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf565282-2490-4e6a-af4b-d85efbf5bab3-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.009456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.009443 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcljb\" (UniqueName: \"kubernetes.io/projected/cf565282-2490-4e6a-af4b-d85efbf5bab3-kube-api-access-gcljb\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.009787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.009763 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf565282-2490-4e6a-af4b-d85efbf5bab3-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.009865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.009805 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf565282-2490-4e6a-af4b-d85efbf5bab3-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.079660 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.079631 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 02 20:21:18.081376 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.081351 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff"] Nov 02 20:21:18.082195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.082178 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcljb\" (UniqueName: \"kubernetes.io/projected/cf565282-2490-4e6a-af4b-d85efbf5bab3-kube-api-access-gcljb\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.090539 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.090516 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf565282-2490-4e6a-af4b-d85efbf5bab3-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-d8fff\" (UID: \"cf565282-2490-4e6a-af4b-d85efbf5bab3\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.203278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.203211 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:18.411172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.411140 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2r66m" Nov 02 20:21:18.612473 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:18.612415 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff"] Nov 02 20:21:18.615097 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:21:18.615066 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf565282_2490_4e6a_af4b_d85efbf5bab3.slice/crio-c54cb4cc18b793703ec391abff6e159bd98b5d9d9cc4431c0e27f97668e3719a WatchSource:0}: Error finding container c54cb4cc18b793703ec391abff6e159bd98b5d9d9cc4431c0e27f97668e3719a: Status 404 returned error can't find the container with id c54cb4cc18b793703ec391abff6e159bd98b5d9d9cc4431c0e27f97668e3719a Nov 02 20:21:19.467415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:19.467374 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" event={"ID":"cf565282-2490-4e6a-af4b-d85efbf5bab3","Type":"ContainerStarted","Data":"c54cb4cc18b793703ec391abff6e159bd98b5d9d9cc4431c0e27f97668e3719a"} Nov 02 20:21:32.656976 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:21:32.656940 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:21:47.616754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:47.616716 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" event={"ID":"cf565282-2490-4e6a-af4b-d85efbf5bab3","Type":"ContainerStarted","Data":"c9ff034b2ee8c55774ca49f2f96f31b2beba85bf05b2cdb3ce5a375d8f808cfd"} Nov 02 20:21:47.617245 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:47.616826 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:21:47.773580 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:21:47.773509 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" podStartSLOduration=2.474079575 podStartE2EDuration="30.773493237s" totalImagesPullingTime="28.299413662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:21:47.773110591 +0000 UTC m=+380.137721795" watchObservedRunningTime="2026-11-02 20:21:47.773493237 +0000 UTC m=+380.138104444" Nov 02 20:22:02.656908 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:22:02.656867 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:22:08.623661 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:08.623622 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-d8fff" Nov 02 20:22:32.656985 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:22:32.656948 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:22:47.678864 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.678825 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5"] Nov 02 20:22:47.684151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.684132 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.753084 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.753058 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-maas-api-tls\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.753228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.753100 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcb2t\" (UniqueName: \"kubernetes.io/projected/167471f6-0c2a-499b-883b-a1bb8aadc48b-kube-api-access-hcb2t\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.753228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.753188 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-metrics-certs\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.753301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.753227 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167471f6-0c2a-499b-883b-a1bb8aadc48b-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.780709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.778891 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bg8z4\"" Nov 02 20:22:47.780709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.778952 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:22:47.780914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.779078 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 20:22:47.780914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.779157 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:22:47.780914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.779662 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 20:22:47.781150 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.781131 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5"] Nov 02 20:22:47.854021 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.853983 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-metrics-certs\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.854207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.854028 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167471f6-0c2a-499b-883b-a1bb8aadc48b-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.854207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.854130 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-maas-api-tls\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.854207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.854170 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcb2t\" (UniqueName: \"kubernetes.io/projected/167471f6-0c2a-499b-883b-a1bb8aadc48b-kube-api-access-hcb2t\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.854936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.854914 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167471f6-0c2a-499b-883b-a1bb8aadc48b-openshift-service-ca\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.856401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.856376 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-maas-api-tls\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.856607 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.856587 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-metrics-certs\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.911031 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.911008 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcb2t\" (UniqueName: \"kubernetes.io/projected/167471f6-0c2a-499b-883b-a1bb8aadc48b-kube-api-access-hcb2t\") pod \"maas-api-6d669c7fcd-ccrj5\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:47.996014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:47.995942 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:22:48.361194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:48.361166 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5"] Nov 02 20:22:48.364300 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:22:48.364269 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167471f6_0c2a_499b_883b_a1bb8aadc48b.slice/crio-0d99b5b64fafcbf1186ee6791db3f0d79c9814e1184cbc3e6343ad604eb2a7be WatchSource:0}: Error finding container 0d99b5b64fafcbf1186ee6791db3f0d79c9814e1184cbc3e6343ad604eb2a7be: Status 404 returned error can't find the container with id 0d99b5b64fafcbf1186ee6791db3f0d79c9814e1184cbc3e6343ad604eb2a7be Nov 02 20:22:48.904679 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:48.904633 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" event={"ID":"167471f6-0c2a-499b-883b-a1bb8aadc48b","Type":"ContainerStarted","Data":"0d99b5b64fafcbf1186ee6791db3f0d79c9814e1184cbc3e6343ad604eb2a7be"} Nov 02 20:22:50.808442 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:50.808406 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ntg25"] Nov 02 20:22:50.819709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:50.819656 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:50.867179 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:50.867141 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ntg25"] Nov 02 20:22:50.921358 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:50.921331 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-xd7rc\"" Nov 02 20:22:50.921358 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:50.921340 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 02 20:22:50.984902 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:50.984873 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1389a165-be88-48ce-abd7-17480feb6620-tls-certs\") pod \"model-serving-api-75bd89fdfc-ntg25\" (UID: \"1389a165-be88-48ce-abd7-17480feb6620\") " pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:50.985050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:50.984995 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgftd\" (UniqueName: \"kubernetes.io/projected/1389a165-be88-48ce-abd7-17480feb6620-kube-api-access-dgftd\") pod \"model-serving-api-75bd89fdfc-ntg25\" (UID: \"1389a165-be88-48ce-abd7-17480feb6620\") " pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:51.086402 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:51.086323 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgftd\" (UniqueName: \"kubernetes.io/projected/1389a165-be88-48ce-abd7-17480feb6620-kube-api-access-dgftd\") pod \"model-serving-api-75bd89fdfc-ntg25\" (UID: \"1389a165-be88-48ce-abd7-17480feb6620\") " pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:51.086402 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:51.086369 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1389a165-be88-48ce-abd7-17480feb6620-tls-certs\") pod \"model-serving-api-75bd89fdfc-ntg25\" (UID: \"1389a165-be88-48ce-abd7-17480feb6620\") " pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:51.088716 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:51.088698 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1389a165-be88-48ce-abd7-17480feb6620-tls-certs\") pod \"model-serving-api-75bd89fdfc-ntg25\" (UID: \"1389a165-be88-48ce-abd7-17480feb6620\") " pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:51.143584 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:51.143551 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgftd\" (UniqueName: \"kubernetes.io/projected/1389a165-be88-48ce-abd7-17480feb6620-kube-api-access-dgftd\") pod \"model-serving-api-75bd89fdfc-ntg25\" (UID: \"1389a165-be88-48ce-abd7-17480feb6620\") " pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:51.437865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:51.437785 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:51.800319 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:51.800117 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ntg25"] Nov 02 20:22:51.803217 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:22:51.803191 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1389a165_be88_48ce_abd7_17480feb6620.slice/crio-c1651527ff745328e0b99a22cb9f47922f81e72209c320828e32d0a6e018193a WatchSource:0}: Error finding container c1651527ff745328e0b99a22cb9f47922f81e72209c320828e32d0a6e018193a: Status 404 returned error can't find the container with id c1651527ff745328e0b99a22cb9f47922f81e72209c320828e32d0a6e018193a Nov 02 20:22:51.922405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:51.922372 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" event={"ID":"1389a165-be88-48ce-abd7-17480feb6620","Type":"ContainerStarted","Data":"c1651527ff745328e0b99a22cb9f47922f81e72209c320828e32d0a6e018193a"} Nov 02 20:22:54.945034 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:54.944989 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" event={"ID":"1389a165-be88-48ce-abd7-17480feb6620","Type":"ContainerStarted","Data":"4e68357f1e6a817532ab3952d869c6acefa82ea453515d541d767f6b4122bf61"} Nov 02 20:22:54.945717 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:54.945692 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:22:55.054344 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:22:55.054291 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" podStartSLOduration=2.534119907 podStartE2EDuration="5.054276782s" totalImagesPullingTime="2.520156875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:22:55.052932432 +0000 UTC m=+447.417543635" watchObservedRunningTime="2026-11-02 20:22:55.054276782 +0000 UTC m=+447.418888016" Nov 02 20:23:00.977226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:00.977192 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" event={"ID":"167471f6-0c2a-499b-883b-a1bb8aadc48b","Type":"ContainerStarted","Data":"e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663"} Nov 02 20:23:00.977653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:00.977286 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:23:01.088307 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:01.088257 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" podStartSLOduration=2.08251825 podStartE2EDuration="14.088243617s" totalImagesPullingTime="12.005725367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:23:01.086806936 +0000 UTC m=+453.451418142" watchObservedRunningTime="2026-11-02 20:23:01.088243617 +0000 UTC m=+453.452854821" Nov 02 20:23:01.984174 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:01.984142 5420 generic.go:356] "Generic (PLEG): container finished" podID="eccd3c31-d213-4763-b79b-ee970b2c334d" containerID="f05f20118cc98775f57607ea40da9ac9920c44b93e81416f6e0544fbc0d8b65d" exitCode=0 Nov 02 20:23:01.984637 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:01.984221 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" event={"ID":"eccd3c31-d213-4763-b79b-ee970b2c334d","Type":"ContainerDied","Data":"f05f20118cc98775f57607ea40da9ac9920c44b93e81416f6e0544fbc0d8b65d"} Nov 02 20:23:01.984707 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:01.984698 5420 scope.go:122] "RemoveContainer" containerID="f05f20118cc98775f57607ea40da9ac9920c44b93e81416f6e0544fbc0d8b65d" Nov 02 20:23:01.985687 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:01.985645 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:23:01.985754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:01.985727 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:23:01.985792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:01.985759 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:23:02.669695 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:23:02.664884 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:23:02.852306 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:02.852275 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:23:02.991584 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:02.991492 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" event={"ID":"eccd3c31-d213-4763-b79b-ee970b2c334d","Type":"ContainerStarted","Data":"4bbc16eb54d0ba5d2a77bcaae507767242bc83dd9351ff06e20ede90ff4e46d8"} Nov 02 20:23:03.405529 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:03.405498 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:23:03.406751 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:03.406723 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8vqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 02 20:23:03.406882 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:03.406771 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" podUID="eccd3c31-d213-4763-b79b-ee970b2c334d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 02 20:23:04.405616 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:04.405578 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8vqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 02 20:23:04.406046 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:04.405632 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" podUID="eccd3c31-d213-4763-b79b-ee970b2c334d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 02 20:23:05.405217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:05.405183 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8vqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 02 20:23:05.405391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:05.405243 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" podUID="eccd3c31-d213-4763-b79b-ee970b2c334d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 02 20:23:06.405126 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:06.405090 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8vqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 02 20:23:06.405511 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:06.405145 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" podUID="eccd3c31-d213-4763-b79b-ee970b2c334d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 02 20:23:06.982871 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:06.982838 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-ntg25" Nov 02 20:23:06.990299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:06.990274 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:23:07.405016 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:07.404970 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8vqkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 02 20:23:07.405179 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:07.405035 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" podUID="eccd3c31-d213-4763-b79b-ee970b2c334d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 02 20:23:08.409478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:08.409407 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:23:08.409896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:08.409680 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:23:08.410367 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:08.410346 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" Nov 02 20:23:18.091131 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.091097 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5"] Nov 02 20:23:18.091618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.091288 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" podUID="167471f6-0c2a-499b-883b-a1bb8aadc48b" containerName="maas-api" containerID="cri-o://e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663" gracePeriod=30 Nov 02 20:23:18.330522 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.330496 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:23:18.425342 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.425273 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-maas-api-tls\") pod \"167471f6-0c2a-499b-883b-a1bb8aadc48b\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " Nov 02 20:23:18.425342 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.425328 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167471f6-0c2a-499b-883b-a1bb8aadc48b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167471f6-0c2a-499b-883b-a1bb8aadc48b-openshift-service-ca\") pod \"167471f6-0c2a-499b-883b-a1bb8aadc48b\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " Nov 02 20:23:18.425506 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.425352 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-metrics-certs\") pod \"167471f6-0c2a-499b-883b-a1bb8aadc48b\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " Nov 02 20:23:18.425506 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.425406 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167471f6-0c2a-499b-883b-a1bb8aadc48b-kube-api-access-hcb2t\" (UniqueName: \"kubernetes.io/projected/167471f6-0c2a-499b-883b-a1bb8aadc48b-kube-api-access-hcb2t\") pod \"167471f6-0c2a-499b-883b-a1bb8aadc48b\" (UID: \"167471f6-0c2a-499b-883b-a1bb8aadc48b\") " Nov 02 20:23:18.425760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.425734 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167471f6-0c2a-499b-883b-a1bb8aadc48b-openshift-service-ca" pod "167471f6-0c2a-499b-883b-a1bb8aadc48b" (UID: "167471f6-0c2a-499b-883b-a1bb8aadc48b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:23:18.427480 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.427455 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167471f6-0c2a-499b-883b-a1bb8aadc48b-kube-api-access-hcb2t" pod "167471f6-0c2a-499b-883b-a1bb8aadc48b" (UID: "167471f6-0c2a-499b-883b-a1bb8aadc48b"). InnerVolumeSpecName "kube-api-access-hcb2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:23:18.427578 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.427484 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-maas-api-tls" pod "167471f6-0c2a-499b-883b-a1bb8aadc48b" (UID: "167471f6-0c2a-499b-883b-a1bb8aadc48b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:18.427629 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.427574 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-metrics-certs" pod "167471f6-0c2a-499b-883b-a1bb8aadc48b" (UID: "167471f6-0c2a-499b-883b-a1bb8aadc48b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:23:18.526052 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.526019 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/167471f6-0c2a-499b-883b-a1bb8aadc48b-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:18.526052 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.526047 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:18.526052 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.526055 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcb2t\" (UniqueName: \"kubernetes.io/projected/167471f6-0c2a-499b-883b-a1bb8aadc48b-kube-api-access-hcb2t\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:18.526275 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:18.526064 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/167471f6-0c2a-499b-883b-a1bb8aadc48b-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:23:19.070525 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.070487 5420 generic.go:356] "Generic (PLEG): container finished" podID="167471f6-0c2a-499b-883b-a1bb8aadc48b" containerID="e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663" exitCode=0 Nov 02 20:23:19.070525 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.070524 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" Nov 02 20:23:19.070932 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.070608 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" event={"ID":"167471f6-0c2a-499b-883b-a1bb8aadc48b","Type":"ContainerDied","Data":"e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663"} Nov 02 20:23:19.070932 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.070691 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5" event={"ID":"167471f6-0c2a-499b-883b-a1bb8aadc48b","Type":"ContainerDied","Data":"0d99b5b64fafcbf1186ee6791db3f0d79c9814e1184cbc3e6343ad604eb2a7be"} Nov 02 20:23:19.070932 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.070723 5420 scope.go:122] "RemoveContainer" containerID="e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663" Nov 02 20:23:19.080285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.080270 5420 scope.go:122] "RemoveContainer" containerID="e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663" Nov 02 20:23:19.080599 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:23:19.080582 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663\": container with ID starting with e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663 not found: ID does not exist" containerID="e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663" Nov 02 20:23:19.080658 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.080607 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663"} err="failed to get container status \"e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663\": rpc error: code = NotFound desc = could not find container \"e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663\": container with ID starting with e866cff99185b7088e5bbfad650a1fd7f3908d37998094a632beb159a538e663 not found: ID does not exist" Nov 02 20:23:19.226826 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.226790 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5"] Nov 02 20:23:19.230562 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.230534 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-ccrj5"] Nov 02 20:23:19.772918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:23:19.772889 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167471f6-0c2a-499b-883b-a1bb8aadc48b" path="/var/lib/kubelet/pods/167471f6-0c2a-499b-883b-a1bb8aadc48b/volumes" Nov 02 20:23:32.657012 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:23:32.656968 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:24:02.657070 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:24:02.657025 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:24:32.657168 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:24:32.657130 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:25:02.656803 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:25:02.656726 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:25:05.063490 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.063459 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj"] Nov 02 20:25:05.063940 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.063830 5420 cpu_manager.go:414] "containerMap: removing container" podUID="167471f6-0c2a-499b-883b-a1bb8aadc48b" containerName="maas-api" Nov 02 20:25:05.063940 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.063842 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167471f6-0c2a-499b-883b-a1bb8aadc48b" containerName="maas-api" Nov 02 20:25:05.063940 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.063911 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="167471f6-0c2a-499b-883b-a1bb8aadc48b" containerName="maas-api" Nov 02 20:25:05.067097 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.067074 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.111972 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.111945 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.112127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.111983 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.112127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.112010 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.112127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.112081 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.112127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.112115 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774962ca-647f-40c4-855b-157ab23abc51-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.112275 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.112219 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8pk\" (UniqueName: \"kubernetes.io/projected/774962ca-647f-40c4-855b-157ab23abc51-kube-api-access-2m8pk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.140918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.140888 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 20:25:05.161883 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.161853 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 20:25:05.161883 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.161853 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rwxhc\"" Nov 02 20:25:05.162060 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.161951 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 20:25:05.163849 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.163831 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj"] Nov 02 20:25:05.213317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213292 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.213476 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213324 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.213476 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213346 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.213476 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213364 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.213476 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213380 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774962ca-647f-40c4-855b-157ab23abc51-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.213476 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213421 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8pk\" (UniqueName: \"kubernetes.io/projected/774962ca-647f-40c4-855b-157ab23abc51-kube-api-access-2m8pk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.213805 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213785 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.213869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213819 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.213869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.213863 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.215717 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.215697 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/774962ca-647f-40c4-855b-157ab23abc51-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.215883 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.215864 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/774962ca-647f-40c4-855b-157ab23abc51-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.270331 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.270302 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8pk\" (UniqueName: \"kubernetes.io/projected/774962ca-647f-40c4-855b-157ab23abc51-kube-api-access-2m8pk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj\" (UID: \"774962ca-647f-40c4-855b-157ab23abc51\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.379215 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.379139 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:05.743309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:05.743216 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj"] Nov 02 20:25:05.750104 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:25:05.750073 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774962ca_647f_40c4_855b_157ab23abc51.slice/crio-ae99414672084c0e3bd83894f0b65e0a094972e1784e342975393bfd58cd427a WatchSource:0}: Error finding container ae99414672084c0e3bd83894f0b65e0a094972e1784e342975393bfd58cd427a: Status 404 returned error can't find the container with id ae99414672084c0e3bd83894f0b65e0a094972e1784e342975393bfd58cd427a Nov 02 20:25:06.599816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:06.599780 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" event={"ID":"774962ca-647f-40c4-855b-157ab23abc51","Type":"ContainerStarted","Data":"ae99414672084c0e3bd83894f0b65e0a094972e1784e342975393bfd58cd427a"} Nov 02 20:25:08.612198 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:08.612157 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" event={"ID":"774962ca-647f-40c4-855b-157ab23abc51","Type":"ContainerStarted","Data":"84fcaedabd417f04d68e45647a307ede13fe12f38717b35759171f372c7632e0"} Nov 02 20:25:08.723863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:08.723769 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" podStartSLOduration=1.6164127449999999 podStartE2EDuration="3.723756183s" totalImagesPullingTime="2.107343438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:08.722384664 +0000 UTC m=+581.086995867" watchObservedRunningTime="2026-11-02 20:25:08.723756183 +0000 UTC m=+581.088367438" Nov 02 20:25:09.380178 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:09.380144 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:12.361324 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.361287 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4"] Nov 02 20:25:12.364900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.364865 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.413506 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.413481 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 20:25:12.461241 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.461211 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4"] Nov 02 20:25:12.474513 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.474484 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.474684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.474525 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.474684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.474558 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.474775 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.474692 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.474775 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.474737 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d276f839-35ce-41a4-aa67-f46d4d405f3f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.474851 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.474775 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fz8l\" (UniqueName: \"kubernetes.io/projected/d276f839-35ce-41a4-aa67-f46d4d405f3f-kube-api-access-6fz8l\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.575311 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.575273 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.575464 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.575338 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.575464 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.575371 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.575464 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.575416 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d276f839-35ce-41a4-aa67-f46d4d405f3f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.575644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.575463 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fz8l\" (UniqueName: \"kubernetes.io/projected/d276f839-35ce-41a4-aa67-f46d4d405f3f-kube-api-access-6fz8l\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.575644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.575502 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.575856 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.575828 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.576134 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.576110 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.576232 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.576167 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.578086 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.578059 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d276f839-35ce-41a4-aa67-f46d4d405f3f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.578583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.578559 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d276f839-35ce-41a4-aa67-f46d4d405f3f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.632273 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.632207 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fz8l\" (UniqueName: \"kubernetes.io/projected/d276f839-35ce-41a4-aa67-f46d4d405f3f-kube-api-access-6fz8l\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-z8sv4\" (UID: \"d276f839-35ce-41a4-aa67-f46d4d405f3f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:12.679382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:12.679349 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:13.046949 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:13.046925 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4"] Nov 02 20:25:13.049491 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:25:13.049453 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd276f839_35ce_41a4_aa67_f46d4d405f3f.slice/crio-80382e5cc3a4015ddab0e5cb47d2181c1e23f60d51d22d4b8639251f7a0886fa WatchSource:0}: Error finding container 80382e5cc3a4015ddab0e5cb47d2181c1e23f60d51d22d4b8639251f7a0886fa: Status 404 returned error can't find the container with id 80382e5cc3a4015ddab0e5cb47d2181c1e23f60d51d22d4b8639251f7a0886fa Nov 02 20:25:13.636572 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:13.636535 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" event={"ID":"d276f839-35ce-41a4-aa67-f46d4d405f3f","Type":"ContainerStarted","Data":"80382e5cc3a4015ddab0e5cb47d2181c1e23f60d51d22d4b8639251f7a0886fa"} Nov 02 20:25:14.642977 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:14.642937 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" event={"ID":"d276f839-35ce-41a4-aa67-f46d4d405f3f","Type":"ContainerStarted","Data":"989fa0b3f12d7bcc8bf56e091b867bcf78efe52e30963dd26040f9a417ab9d9e"} Nov 02 20:25:14.680371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:14.680339 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:14.754762 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:14.754712 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" podStartSLOduration=2.135827243 podStartE2EDuration="2.754692778s" totalImagesPullingTime="618.865535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:14.752861975 +0000 UTC m=+587.117473181" watchObservedRunningTime="2026-11-02 20:25:14.754692778 +0000 UTC m=+587.119303980" Nov 02 20:25:15.379486 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:15.379454 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:15.391589 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:15.391565 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:15.658398 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:15.658305 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj" Nov 02 20:25:22.679741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:22.679703 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:22.692279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:22.692254 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:23.045427 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.045351 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8"] Nov 02 20:25:23.048943 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.048922 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.097772 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.097737 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 20:25:23.104516 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.104489 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8"] Nov 02 20:25:23.184041 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.183989 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.184230 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.184161 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.184286 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.184238 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6qj\" (UniqueName: \"kubernetes.io/projected/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-kube-api-access-8b6qj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.184286 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.184264 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.184391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.184287 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.184391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.184313 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.285828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.285777 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.286023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.285851 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.286023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.285892 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6qj\" (UniqueName: \"kubernetes.io/projected/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-kube-api-access-8b6qj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.286023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.285917 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.286023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.285946 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.286023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.285972 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.286269 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.286243 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.286329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.286288 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.286401 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.286377 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.288140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.288118 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.288392 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.288377 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.343451 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.343422 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6qj\" (UniqueName: \"kubernetes.io/projected/3d09f5c8-eea4-44f5-8c9b-3643e053f0eb-kube-api-access-8b6qj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8\" (UID: \"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.361839 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.361809 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:23.691729 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.691636 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-z8sv4" Nov 02 20:25:23.727201 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:23.727174 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8"] Nov 02 20:25:23.730047 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:25:23.730022 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d09f5c8_eea4_44f5_8c9b_3643e053f0eb.slice/crio-f1bb691d95cb2f8bb9b69f8df3202a7186c3a437239c0f06485251dd88b4da01 WatchSource:0}: Error finding container f1bb691d95cb2f8bb9b69f8df3202a7186c3a437239c0f06485251dd88b4da01: Status 404 returned error can't find the container with id f1bb691d95cb2f8bb9b69f8df3202a7186c3a437239c0f06485251dd88b4da01 Nov 02 20:25:24.694580 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:24.694494 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" event={"ID":"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb","Type":"ContainerStarted","Data":"34473837f14b84b0b5d2719e6bdce46a6dcfe45347c0cf453a79304b4b12d7cd"} Nov 02 20:25:24.694580 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:24.694555 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" event={"ID":"3d09f5c8-eea4-44f5-8c9b-3643e053f0eb","Type":"ContainerStarted","Data":"f1bb691d95cb2f8bb9b69f8df3202a7186c3a437239c0f06485251dd88b4da01"} Nov 02 20:25:24.805990 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:24.805936 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" podStartSLOduration=1.169087031 podStartE2EDuration="1.805923343s" totalImagesPullingTime="636.836312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:24.803813564 +0000 UTC m=+597.168424768" watchObservedRunningTime="2026-11-02 20:25:24.805923343 +0000 UTC m=+597.170534588" Nov 02 20:25:25.362190 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:25.362154 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:26.065762 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.065727 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-x5gcv"] Nov 02 20:25:26.069344 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.069321 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.164099 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.164057 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 20:25:26.166301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.166272 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-x5gcv"] Nov 02 20:25:26.211573 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.211535 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-home\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.211772 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.211651 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s28k\" (UniqueName: \"kubernetes.io/projected/fea054c8-a620-453c-862a-8ce7eeed9e0c-kube-api-access-9s28k\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.211846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.211820 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.211897 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.211884 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.211941 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.211927 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.211986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.211948 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea054c8-a620-453c-862a-8ce7eeed9e0c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313036 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313135 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313170 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313190 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea054c8-a620-453c-862a-8ce7eeed9e0c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313252 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-home\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313358 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s28k\" (UniqueName: \"kubernetes.io/projected/fea054c8-a620-453c-862a-8ce7eeed9e0c-kube-api-access-9s28k\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313494 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313628 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313541 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.313749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.313730 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-home\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.315590 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.315570 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fea054c8-a620-453c-862a-8ce7eeed9e0c-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.315751 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.315733 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fea054c8-a620-453c-862a-8ce7eeed9e0c-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.370371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.370298 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s28k\" (UniqueName: \"kubernetes.io/projected/fea054c8-a620-453c-862a-8ce7eeed9e0c-kube-api-access-9s28k\") pod \"echo-simulator-kserve-64b6f8f96c-x5gcv\" (UID: \"fea054c8-a620-453c-862a-8ce7eeed9e0c\") " pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.383144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.383107 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:26.750444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:26.750241 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-x5gcv"] Nov 02 20:25:26.753443 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:25:26.753411 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea054c8_a620_453c_862a_8ce7eeed9e0c.slice/crio-37499c0a7c67e1ca98b9d9c98f7280e44a1303105fde713c07e232e5b2f48e3e WatchSource:0}: Error finding container 37499c0a7c67e1ca98b9d9c98f7280e44a1303105fde713c07e232e5b2f48e3e: Status 404 returned error can't find the container with id 37499c0a7c67e1ca98b9d9c98f7280e44a1303105fde713c07e232e5b2f48e3e Nov 02 20:25:27.726081 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:27.726042 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" event={"ID":"fea054c8-a620-453c-862a-8ce7eeed9e0c","Type":"ContainerStarted","Data":"37499c0a7c67e1ca98b9d9c98f7280e44a1303105fde713c07e232e5b2f48e3e"} Nov 02 20:25:29.739370 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:29.739331 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" event={"ID":"fea054c8-a620-453c-862a-8ce7eeed9e0c","Type":"ContainerStarted","Data":"3b78f8f087b2f8a42061fb64e2bb521fea90aaafd5aad5c58cc0d79a37eaecdc"} Nov 02 20:25:29.850133 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:29.850086 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" podStartSLOduration=0.99139977 podStartE2EDuration="3.850071434s" totalImagesPullingTime="2.858671664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:29.848302457 +0000 UTC m=+602.212913663" watchObservedRunningTime="2026-11-02 20:25:29.850071434 +0000 UTC m=+602.214682643" Nov 02 20:25:30.384108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:30.384072 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:32.658130 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:25:32.658089 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:25:33.362065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:33.362035 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:33.373963 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:33.373936 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:33.776598 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:33.776510 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8" Nov 02 20:25:36.383912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:36.383876 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:36.396207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:36.396183 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:36.786591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:36.786510 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-x5gcv" Nov 02 20:25:38.366614 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.366580 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh"] Nov 02 20:25:38.370500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.370479 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526318 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526282 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526492 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526328 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526492 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526444 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526492 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526473 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b91fab4a-d096-46f5-b4b4-6e912edc7785-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526606 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526548 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526606 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526578 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526704 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526651 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526745 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526702 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.526745 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.526731 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjxn\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-kube-api-access-7cjxn\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.564525 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.564496 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh"] Nov 02 20:25:38.627553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627463 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.627553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627527 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.627828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627560 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.627828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627587 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjxn\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-kube-api-access-7cjxn\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.627828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627665 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.627828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627753 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.627828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627795 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.627828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627824 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b91fab4a-d096-46f5-b4b4-6e912edc7785-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.628123 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.627898 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.628180 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.628113 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.628233 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.628187 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.628270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.628232 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.628384 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.628359 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.628698 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.628651 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b91fab4a-d096-46f5-b4b4-6e912edc7785-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.630349 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.630325 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.630665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.630647 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.684571 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.684541 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.684908 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.684888 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjxn\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-kube-api-access-7cjxn\") pod \"maas-default-gateway-openshift-default-85544d4cd7-swdjh\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:38.983179 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:38.983093 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:39.400327 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:39.400283 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh"] Nov 02 20:25:39.434624 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:25:39.434590 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91fab4a_d096_46f5_b4b4_6e912edc7785.slice/crio-ab62a8d97eaf650e39e325dd8074da88ae400050e3444d4442d2fb931f5b731d WatchSource:0}: Error finding container ab62a8d97eaf650e39e325dd8074da88ae400050e3444d4442d2fb931f5b731d: Status 404 returned error can't find the container with id ab62a8d97eaf650e39e325dd8074da88ae400050e3444d4442d2fb931f5b731d Nov 02 20:25:39.436746 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:39.436716 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:39.436816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:39.436802 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:39.436854 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:39.436832 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:25:39.796007 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:39.795914 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" event={"ID":"b91fab4a-d096-46f5-b4b4-6e912edc7785","Type":"ContainerStarted","Data":"0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7"} Nov 02 20:25:39.796274 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:39.796248 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" event={"ID":"b91fab4a-d096-46f5-b4b4-6e912edc7785","Type":"ContainerStarted","Data":"ab62a8d97eaf650e39e325dd8074da88ae400050e3444d4442d2fb931f5b731d"} Nov 02 20:25:39.909932 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:39.909872 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" podStartSLOduration=1.909851213 podStartE2EDuration="1.909851213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:25:39.907973214 +0000 UTC m=+612.272584418" watchObservedRunningTime="2026-11-02 20:25:39.909851213 +0000 UTC m=+612.274462420" Nov 02 20:25:39.986823 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:39.986780 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:40.985726 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:40.984953 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-swdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 20:25:40.985726 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:40.985028 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 20:25:41.985169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:41.985132 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-swdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 20:25:41.985481 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:41.985449 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 20:25:42.984282 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:42.984244 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-swdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 20:25:42.984795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:42.984320 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 20:25:43.984187 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:43.984139 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-swdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 20:25:43.984406 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:43.984211 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 20:25:44.984244 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:44.984195 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-swdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 20:25:44.984450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:44.984267 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 20:25:45.983960 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:45.983930 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-swdjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 02 20:25:45.984135 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:45.983985 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 02 20:25:47.579366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:47.579333 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:47.579963 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:47.579939 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:47.580598 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:47.580579 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:25:58.026524 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.026483 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-29skg"] Nov 02 20:25:58.032044 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.032021 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:25:58.085522 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.085489 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-29skg"] Nov 02 20:25:58.129489 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.129462 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 20:25:58.129489 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.129493 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 20:25:58.129707 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.129553 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-ckqmt\"" Nov 02 20:25:58.135087 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.135063 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cce5518-fee3-47c2-9586-e16f88f5e109-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-29skg\" (UID: \"2cce5518-fee3-47c2-9586-e16f88f5e109\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:25:58.135191 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.135176 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krqq\" (UniqueName: \"kubernetes.io/projected/2cce5518-fee3-47c2-9586-e16f88f5e109-kube-api-access-4krqq\") pod \"ai-gateway-controller-7d6c696b49-29skg\" (UID: \"2cce5518-fee3-47c2-9586-e16f88f5e109\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:25:58.235904 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.235869 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krqq\" (UniqueName: \"kubernetes.io/projected/2cce5518-fee3-47c2-9586-e16f88f5e109-kube-api-access-4krqq\") pod \"ai-gateway-controller-7d6c696b49-29skg\" (UID: \"2cce5518-fee3-47c2-9586-e16f88f5e109\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:25:58.236097 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.235963 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cce5518-fee3-47c2-9586-e16f88f5e109-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-29skg\" (UID: \"2cce5518-fee3-47c2-9586-e16f88f5e109\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:25:58.238457 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.238433 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cce5518-fee3-47c2-9586-e16f88f5e109-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-29skg\" (UID: \"2cce5518-fee3-47c2-9586-e16f88f5e109\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:25:58.293243 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.293156 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krqq\" (UniqueName: \"kubernetes.io/projected/2cce5518-fee3-47c2-9586-e16f88f5e109-kube-api-access-4krqq\") pod \"ai-gateway-controller-7d6c696b49-29skg\" (UID: \"2cce5518-fee3-47c2-9586-e16f88f5e109\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:25:58.345541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.345502 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:25:58.716146 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.714634 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-29skg"] Nov 02 20:25:58.720310 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:25:58.720277 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cce5518_fee3_47c2_9586_e16f88f5e109.slice/crio-44d8103dbd12c3d07f26ad13431032d5e0f95cef39da41aa709eec155e22db8c WatchSource:0}: Error finding container 44d8103dbd12c3d07f26ad13431032d5e0f95cef39da41aa709eec155e22db8c: Status 404 returned error can't find the container with id 44d8103dbd12c3d07f26ad13431032d5e0f95cef39da41aa709eec155e22db8c Nov 02 20:25:58.722205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.722169 5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:25:58.894873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:25:58.894838 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" event={"ID":"2cce5518-fee3-47c2-9586-e16f88f5e109","Type":"ContainerStarted","Data":"44d8103dbd12c3d07f26ad13431032d5e0f95cef39da41aa709eec155e22db8c"} Nov 02 20:26:02.657654 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:26:02.657615 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:26:04.930896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:04.930810 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" event={"ID":"2cce5518-fee3-47c2-9586-e16f88f5e109","Type":"ContainerStarted","Data":"240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a"} Nov 02 20:26:04.931271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:04.930973 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:26:05.040489 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:05.040437 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" podStartSLOduration=2.206677563 podStartE2EDuration="8.040424202s" totalImagesPullingTime="5.833746639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:05.038959729 +0000 UTC m=+637.403570937" watchObservedRunningTime="2026-11-02 20:26:05.040424202 +0000 UTC m=+637.405035406" Nov 02 20:26:07.773689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.773645 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw"] Nov 02 20:26:07.777277 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.777254 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:07.826493 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.826458 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 20:26:07.901473 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.901437 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lhc87\"" Nov 02 20:26:07.904869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.904834 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw"] Nov 02 20:26:07.943781 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.943741 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5kc\" (UniqueName: \"kubernetes.io/projected/eba2df43-5fb9-4e11-9bea-b7c793484821-kube-api-access-2l5kc\") pod \"payload-pre-processing-7b98bb678c-bmkzw\" (UID: \"eba2df43-5fb9-4e11-9bea-b7c793484821\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:07.943974 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.943878 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba2df43-5fb9-4e11-9bea-b7c793484821-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-bmkzw\" (UID: \"eba2df43-5fb9-4e11-9bea-b7c793484821\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:07.950196 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.950161 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-6g7tz"] Nov 02 20:26:07.953966 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:07.953945 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:08.009355 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.009296 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-6g7tz"] Nov 02 20:26:08.044703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.044582 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph7n8\" (UniqueName: \"kubernetes.io/projected/b7165f85-4e19-4e3a-b333-cffb9d791cf6-kube-api-access-ph7n8\") pod \"payload-processing-7bb9489676-6g7tz\" (UID: \"b7165f85-4e19-4e3a-b333-cffb9d791cf6\") " pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:08.044703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.044625 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5kc\" (UniqueName: \"kubernetes.io/projected/eba2df43-5fb9-4e11-9bea-b7c793484821-kube-api-access-2l5kc\") pod \"payload-pre-processing-7b98bb678c-bmkzw\" (UID: \"eba2df43-5fb9-4e11-9bea-b7c793484821\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:08.044703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.044644 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7165f85-4e19-4e3a-b333-cffb9d791cf6-plugins-config-volume\") pod \"payload-processing-7bb9489676-6g7tz\" (UID: \"b7165f85-4e19-4e3a-b333-cffb9d791cf6\") " pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:08.044943 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.044813 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba2df43-5fb9-4e11-9bea-b7c793484821-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-bmkzw\" (UID: \"eba2df43-5fb9-4e11-9bea-b7c793484821\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:08.045463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.045446 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba2df43-5fb9-4e11-9bea-b7c793484821-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-bmkzw\" (UID: \"eba2df43-5fb9-4e11-9bea-b7c793484821\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:08.101995 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.101969 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5kc\" (UniqueName: \"kubernetes.io/projected/eba2df43-5fb9-4e11-9bea-b7c793484821-kube-api-access-2l5kc\") pod \"payload-pre-processing-7b98bb678c-bmkzw\" (UID: \"eba2df43-5fb9-4e11-9bea-b7c793484821\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:08.146295 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.146253 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph7n8\" (UniqueName: \"kubernetes.io/projected/b7165f85-4e19-4e3a-b333-cffb9d791cf6-kube-api-access-ph7n8\") pod \"payload-processing-7bb9489676-6g7tz\" (UID: \"b7165f85-4e19-4e3a-b333-cffb9d791cf6\") " pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:08.146502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.146309 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7165f85-4e19-4e3a-b333-cffb9d791cf6-plugins-config-volume\") pod \"payload-processing-7bb9489676-6g7tz\" (UID: \"b7165f85-4e19-4e3a-b333-cffb9d791cf6\") " pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:08.147306 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.147280 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7165f85-4e19-4e3a-b333-cffb9d791cf6-plugins-config-volume\") pod \"payload-processing-7bb9489676-6g7tz\" (UID: \"b7165f85-4e19-4e3a-b333-cffb9d791cf6\") " pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:08.203638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.203600 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph7n8\" (UniqueName: \"kubernetes.io/projected/b7165f85-4e19-4e3a-b333-cffb9d791cf6-kube-api-access-ph7n8\") pod \"payload-processing-7bb9489676-6g7tz\" (UID: \"b7165f85-4e19-4e3a-b333-cffb9d791cf6\") " pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:08.271698 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.271579 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:08.391543 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.391508 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:08.689196 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.689150 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-6g7tz"] Nov 02 20:26:08.692305 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:26:08.692274 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7165f85_4e19_4e3a_b333_cffb9d791cf6.slice/crio-10343122838a9f34a5d970c6fa39ec0a7fb1ed3d395f31183ae86c3783123a46 WatchSource:0}: Error finding container 10343122838a9f34a5d970c6fa39ec0a7fb1ed3d395f31183ae86c3783123a46: Status 404 returned error can't find the container with id 10343122838a9f34a5d970c6fa39ec0a7fb1ed3d395f31183ae86c3783123a46 Nov 02 20:26:08.771521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.771482 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw"] Nov 02 20:26:08.774467 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:26:08.774443 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba2df43_5fb9_4e11_9bea_b7c793484821.slice/crio-907b56141355a1bf36734f700488cbaa312de13f783df1727c0b84829d909181 WatchSource:0}: Error finding container 907b56141355a1bf36734f700488cbaa312de13f783df1727c0b84829d909181: Status 404 returned error can't find the container with id 907b56141355a1bf36734f700488cbaa312de13f783df1727c0b84829d909181 Nov 02 20:26:08.955817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.955718 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" event={"ID":"eba2df43-5fb9-4e11-9bea-b7c793484821","Type":"ContainerStarted","Data":"907b56141355a1bf36734f700488cbaa312de13f783df1727c0b84829d909181"} Nov 02 20:26:08.956896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:08.956873 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" event={"ID":"b7165f85-4e19-4e3a-b333-cffb9d791cf6","Type":"ContainerStarted","Data":"10343122838a9f34a5d970c6fa39ec0a7fb1ed3d395f31183ae86c3783123a46"} Nov 02 20:26:11.978804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:11.978706 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" event={"ID":"eba2df43-5fb9-4e11-9bea-b7c793484821","Type":"ContainerStarted","Data":"680d66ab9fee82393a604503fe6483c1fa815b8aaee90dd5d9155d008d49ef13"} Nov 02 20:26:11.978804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:11.978775 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:11.980492 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:11.980470 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" event={"ID":"b7165f85-4e19-4e3a-b333-cffb9d791cf6","Type":"ContainerStarted","Data":"c2bc60c34882743e72055bc38fd4cf12952722a7995dee253177a54f696e6476"} Nov 02 20:26:11.980611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:11.980514 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:12.088394 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:12.088335 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" podStartSLOduration=2.206108453 podStartE2EDuration="5.088321142s" totalImagesPullingTime="2.882212689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:12.087919681 +0000 UTC m=+644.452530899" watchObservedRunningTime="2026-11-02 20:26:12.088321142 +0000 UTC m=+644.452932347" Nov 02 20:26:12.196489 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:12.196422 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" podStartSLOduration=2.240836067 podStartE2EDuration="5.196404898s" totalImagesPullingTime="2.955568831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:12.19553202 +0000 UTC m=+644.560143225" watchObservedRunningTime="2026-11-02 20:26:12.196404898 +0000 UTC m=+644.561016103" Nov 02 20:26:15.938044 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:15.938014 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:26:16.274437 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.274360 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-86f597b94c-588d2"] Nov 02 20:26:16.278550 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.278533 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:16.333141 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.333112 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86f597b94c-588d2"] Nov 02 20:26:16.428469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.428437 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76f30471-010d-4c78-8d5c-7b1fef0f956f-metrics-certs\") pod \"ai-gateway-controller-86f597b94c-588d2\" (UID: \"76f30471-010d-4c78-8d5c-7b1fef0f956f\") " pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:16.428469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.428473 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrf5\" (UniqueName: \"kubernetes.io/projected/76f30471-010d-4c78-8d5c-7b1fef0f956f-kube-api-access-ggrf5\") pod \"ai-gateway-controller-86f597b94c-588d2\" (UID: \"76f30471-010d-4c78-8d5c-7b1fef0f956f\") " pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:16.529444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.529359 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76f30471-010d-4c78-8d5c-7b1fef0f956f-metrics-certs\") pod \"ai-gateway-controller-86f597b94c-588d2\" (UID: \"76f30471-010d-4c78-8d5c-7b1fef0f956f\") " pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:16.529444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.529397 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrf5\" (UniqueName: \"kubernetes.io/projected/76f30471-010d-4c78-8d5c-7b1fef0f956f-kube-api-access-ggrf5\") pod \"ai-gateway-controller-86f597b94c-588d2\" (UID: \"76f30471-010d-4c78-8d5c-7b1fef0f956f\") " pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:16.531974 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.531946 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76f30471-010d-4c78-8d5c-7b1fef0f956f-metrics-certs\") pod \"ai-gateway-controller-86f597b94c-588d2\" (UID: \"76f30471-010d-4c78-8d5c-7b1fef0f956f\") " pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:16.586719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.586690 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrf5\" (UniqueName: \"kubernetes.io/projected/76f30471-010d-4c78-8d5c-7b1fef0f956f-kube-api-access-ggrf5\") pod \"ai-gateway-controller-86f597b94c-588d2\" (UID: \"76f30471-010d-4c78-8d5c-7b1fef0f956f\") " pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:16.590555 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.590536 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:16.953509 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:16.953364 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-86f597b94c-588d2"] Nov 02 20:26:16.956333 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:26:16.956293 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f30471_010d_4c78_8d5c_7b1fef0f956f.slice/crio-daf3aaebfdd1798b247344967ee5fc6cfe1aee49dcfd20e7d32973eaeee9f9da WatchSource:0}: Error finding container daf3aaebfdd1798b247344967ee5fc6cfe1aee49dcfd20e7d32973eaeee9f9da: Status 404 returned error can't find the container with id daf3aaebfdd1798b247344967ee5fc6cfe1aee49dcfd20e7d32973eaeee9f9da Nov 02 20:26:17.016077 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:17.016051 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" event={"ID":"76f30471-010d-4c78-8d5c-7b1fef0f956f","Type":"ContainerStarted","Data":"daf3aaebfdd1798b247344967ee5fc6cfe1aee49dcfd20e7d32973eaeee9f9da"} Nov 02 20:26:18.022903 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:18.022868 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" event={"ID":"76f30471-010d-4c78-8d5c-7b1fef0f956f","Type":"ContainerStarted","Data":"09faa012b2e45ebbde915e55747967e17483c39f9786eee45ac406060c49ed22"} Nov 02 20:26:18.023335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:18.022926 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:18.134609 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:18.134496 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" podStartSLOduration=1.210756142 podStartE2EDuration="2.134482121s" totalImagesPullingTime="923.725979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:26:18.13260433 +0000 UTC m=+650.497215534" watchObservedRunningTime="2026-11-02 20:26:18.134482121 +0000 UTC m=+650.499093355" Nov 02 20:26:22.996829 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:22.996794 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-bmkzw" Nov 02 20:26:22.997201 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:22.996867 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-6g7tz" Nov 02 20:26:29.030788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.030760 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-86f597b94c-588d2" Nov 02 20:26:29.187586 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.187551 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-29skg"] Nov 02 20:26:29.188014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.187981 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" podUID="2cce5518-fee3-47c2-9586-e16f88f5e109" containerName="manager" containerID="cri-o://240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a" gracePeriod=30 Nov 02 20:26:29.431591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.431567 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:26:29.542472 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.542445 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cce5518-fee3-47c2-9586-e16f88f5e109-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cce5518-fee3-47c2-9586-e16f88f5e109-metrics-certs\") pod \"2cce5518-fee3-47c2-9586-e16f88f5e109\" (UID: \"2cce5518-fee3-47c2-9586-e16f88f5e109\") " Nov 02 20:26:29.542472 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.542478 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cce5518-fee3-47c2-9586-e16f88f5e109-kube-api-access-4krqq\" (UniqueName: \"kubernetes.io/projected/2cce5518-fee3-47c2-9586-e16f88f5e109-kube-api-access-4krqq\") pod \"2cce5518-fee3-47c2-9586-e16f88f5e109\" (UID: \"2cce5518-fee3-47c2-9586-e16f88f5e109\") " Nov 02 20:26:29.545011 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.544974 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cce5518-fee3-47c2-9586-e16f88f5e109-kube-api-access-4krqq" pod "2cce5518-fee3-47c2-9586-e16f88f5e109" (UID: "2cce5518-fee3-47c2-9586-e16f88f5e109"). InnerVolumeSpecName "kube-api-access-4krqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:26:29.545134 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.545003 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cce5518-fee3-47c2-9586-e16f88f5e109-metrics-certs" pod "2cce5518-fee3-47c2-9586-e16f88f5e109" (UID: "2cce5518-fee3-47c2-9586-e16f88f5e109"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:26:29.644094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.644023 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cce5518-fee3-47c2-9586-e16f88f5e109-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:29.644094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:29.644051 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krqq\" (UniqueName: \"kubernetes.io/projected/2cce5518-fee3-47c2-9586-e16f88f5e109-kube-api-access-4krqq\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:26:30.085056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.085014 5420 generic.go:356] "Generic (PLEG): container finished" podID="2cce5518-fee3-47c2-9586-e16f88f5e109" containerID="240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a" exitCode=0 Nov 02 20:26:30.085502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.085063 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" Nov 02 20:26:30.085502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.085062 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" event={"ID":"2cce5518-fee3-47c2-9586-e16f88f5e109","Type":"ContainerDied","Data":"240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a"} Nov 02 20:26:30.085502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.085200 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-29skg" event={"ID":"2cce5518-fee3-47c2-9586-e16f88f5e109","Type":"ContainerDied","Data":"44d8103dbd12c3d07f26ad13431032d5e0f95cef39da41aa709eec155e22db8c"} Nov 02 20:26:30.085502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.085228 5420 scope.go:122] "RemoveContainer" containerID="240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a" Nov 02 20:26:30.093732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.093715 5420 scope.go:122] "RemoveContainer" containerID="240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a" Nov 02 20:26:30.093977 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:26:30.093960 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a\": container with ID starting with 240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a not found: ID does not exist" containerID="240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a" Nov 02 20:26:30.094052 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.093985 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a"} err="failed to get container status \"240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a\": rpc error: code = NotFound desc = could not find container \"240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a\": container with ID starting with 240e468fbe7bf1f0edb7f5027dc6d9f848e57d1162a4c07da915306f1d26939a not found: ID does not exist" Nov 02 20:26:30.238276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.238243 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-29skg"] Nov 02 20:26:30.284402 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:30.284375 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-29skg"] Nov 02 20:26:31.773076 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:26:31.773044 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cce5518-fee3-47c2-9586-e16f88f5e109" path="/var/lib/kubelet/pods/2cce5518-fee3-47c2-9586-e16f88f5e109/volumes" Nov 02 20:26:32.658813 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:26:32.658771 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:27:01.763883 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.763841 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz"] Nov 02 20:27:01.764458 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.764250 5420 cpu_manager.go:414] "containerMap: removing container" podUID="2cce5518-fee3-47c2-9586-e16f88f5e109" containerName="manager" Nov 02 20:27:01.764458 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.764261 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cce5518-fee3-47c2-9586-e16f88f5e109" containerName="manager" Nov 02 20:27:01.764458 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.764322 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cce5518-fee3-47c2-9586-e16f88f5e109" containerName="manager" Nov 02 20:27:01.767705 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.767662 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.816484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.816452 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-30ee89-openshift-default-dockercfg-xj9qc\"" Nov 02 20:27:01.866006 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.865970 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz"] Nov 02 20:27:01.939118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939087 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.939118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939118 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.939339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939140 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.939339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939203 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.939339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939296 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.939339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939334 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.939583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939367 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.939583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939473 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:01.939583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:01.939568 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwdm\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-kube-api-access-xqwdm\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040102 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040014 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040102 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040082 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040129 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040172 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040195 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040243 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwdm\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-kube-api-access-xqwdm\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040275 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040300 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040332 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040505 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040754 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040804 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040850 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.040918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.040853 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.042534 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.042513 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.042651 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.042635 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.098198 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.098158 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwdm\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-kube-api-access-xqwdm\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.099644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.099608 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.345571 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.345533 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs"] Nov 02 20:27:02.350322 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.350301 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.381612 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.381591 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:02.399651 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.399630 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-bce768-openshift-default-dockercfg-xdqwh\"" Nov 02 20:27:02.417097 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.417073 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs"] Nov 02 20:27:02.545337 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545303 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.545505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545358 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.545505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545433 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.545505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545474 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.545505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545499 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.545696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545578 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.545696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545612 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.545696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545629 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rzns\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-kube-api-access-8rzns\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.545696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.545651 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647102 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647180 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647199 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647228 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647257 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647286 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647329 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647355 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rzns\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-kube-api-access-8rzns\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647387 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.647569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647472 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.648206 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.647590 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.648267 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.648222 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.648371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.648348 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.648461 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.648443 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.650335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.650310 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.650593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.650575 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.660568 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:27:02.660540 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:27:02.674064 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.674042 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4"] Nov 02 20:27:02.678999 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.678983 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.706065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.706038 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.706178 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.706111 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rzns\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-kube-api-access-8rzns\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:02.728344 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.728321 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-61cdc1-openshift-default-dockercfg-4zg5g\"" Nov 02 20:27:02.738632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.738603 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4"] Nov 02 20:27:02.803296 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.803273 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz"] Nov 02 20:27:02.804870 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:02.804838 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c61ef80_a2e5_4319_8c6f_ba79be60bdf3.slice/crio-8a54fd5da083a5c77265b0a7716665d6867eb4498cda4b783e09e2cb1728609d WatchSource:0}: Error finding container 8a54fd5da083a5c77265b0a7716665d6867eb4498cda4b783e09e2cb1728609d: Status 404 returned error can't find the container with id 8a54fd5da083a5c77265b0a7716665d6867eb4498cda4b783e09e2cb1728609d Nov 02 20:27:02.806853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.806826 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:02.806912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.806887 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:02.806954 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.806915 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:02.849797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.849770 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.849891 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.849804 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvtb9\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-kube-api-access-xvtb9\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.849891 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.849870 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.849984 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.849903 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.849984 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.849930 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.849984 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.849951 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.850087 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.850051 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.850128 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.850107 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.850186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.850170 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.951742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.951635 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.951911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.951744 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.951911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.951779 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.951911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.951807 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.951911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.951874 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.951911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.951906 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.952163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.951987 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.952163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.952022 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvtb9\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-kube-api-access-xvtb9\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.952163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.952052 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.952321 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.952169 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.952459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.952400 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.952729 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.952706 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.952899 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.952876 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.952945 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.952900 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.954264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.954242 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.955325 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.955303 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:02.964203 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:02.964179 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:03.012732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.012691 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:03.012912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.012863 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvtb9\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-kube-api-access-xvtb9\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:03.263077 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.262998 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" event={"ID":"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3","Type":"ContainerStarted","Data":"892f37a058392d43b648e78c0ed91a358c46eb9ad26bca11132f5990a2fb029e"} Nov 02 20:27:03.263260 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.263127 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" event={"ID":"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3","Type":"ContainerStarted","Data":"8a54fd5da083a5c77265b0a7716665d6867eb4498cda4b783e09e2cb1728609d"} Nov 02 20:27:03.292526 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.292497 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:03.380208 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.380163 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podStartSLOduration=2.380147491 podStartE2EDuration="2.380147491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:03.379179033 +0000 UTC m=+695.743790255" watchObservedRunningTime="2026-11-02 20:27:03.380147491 +0000 UTC m=+695.744758697" Nov 02 20:27:03.382374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.382347 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:03.389531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.389508 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:03.390818 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.390796 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs"] Nov 02 20:27:03.393202 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:03.393176 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9eb207_3df4_4ba5_acda_4041ef5bc080.slice/crio-9e8a6b193b922793d16125f9374cb4c47062d575231646b00b38da2d66a84145 WatchSource:0}: Error finding container 9e8a6b193b922793d16125f9374cb4c47062d575231646b00b38da2d66a84145: Status 404 returned error can't find the container with id 9e8a6b193b922793d16125f9374cb4c47062d575231646b00b38da2d66a84145 Nov 02 20:27:03.395815 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.395779 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:03.395907 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.395864 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:03.395965 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.395908 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:03.676327 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.676298 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4"] Nov 02 20:27:03.678634 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:03.678606 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b79a65_9be4_4acb_b68b_19dca7de84cd.slice/crio-e42db50a8de7fe5e967ecb8d3c9ab323aafae41bd8a262c01f8921a1348f77ba WatchSource:0}: Error finding container e42db50a8de7fe5e967ecb8d3c9ab323aafae41bd8a262c01f8921a1348f77ba: Status 404 returned error can't find the container with id e42db50a8de7fe5e967ecb8d3c9ab323aafae41bd8a262c01f8921a1348f77ba Nov 02 20:27:03.681156 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.681124 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:03.681247 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.681189 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:03.681247 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:03.681219 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:04.270416 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.270381 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" event={"ID":"fc9eb207-3df4-4ba5-acda-4041ef5bc080","Type":"ContainerStarted","Data":"d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1"} Nov 02 20:27:04.270880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.270439 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" event={"ID":"fc9eb207-3df4-4ba5-acda-4041ef5bc080","Type":"ContainerStarted","Data":"9e8a6b193b922793d16125f9374cb4c47062d575231646b00b38da2d66a84145"} Nov 02 20:27:04.271983 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.271961 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" event={"ID":"a5b79a65-9be4-4acb-b68b-19dca7de84cd","Type":"ContainerStarted","Data":"deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889"} Nov 02 20:27:04.272071 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.272006 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" event={"ID":"a5b79a65-9be4-4acb-b68b-19dca7de84cd","Type":"ContainerStarted","Data":"e42db50a8de7fe5e967ecb8d3c9ab323aafae41bd8a262c01f8921a1348f77ba"} Nov 02 20:27:04.272116 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.272106 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:04.273059 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.273036 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:04.293600 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.293576 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:04.298436 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.298417 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:04.386855 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.386808 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" podStartSLOduration=2.38679768 podStartE2EDuration="2.38679768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:04.386135495 +0000 UTC m=+696.750746699" watchObservedRunningTime="2026-11-02 20:27:04.38679768 +0000 UTC m=+696.751408885" Nov 02 20:27:04.502402 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.501621 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" podStartSLOduration=2.501602741 podStartE2EDuration="2.501602741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:04.499635554 +0000 UTC m=+696.864246762" watchObservedRunningTime="2026-11-02 20:27:04.501602741 +0000 UTC m=+696.866213952" Nov 02 20:27:04.964597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.964561 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:04.969456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:04.969436 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:05.278036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:05.277958 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:05.278036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:05.278014 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:05.278496 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:05.278431 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:05.278646 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:05.278631 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:11.799129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.799101 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb"] Nov 02 20:27:11.802665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.802644 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834031 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834008 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834180 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834041 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834180 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834089 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkknn\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-kube-api-access-nkknn\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834319 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834232 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834319 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834259 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834319 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834280 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834481 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834515 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.834558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.834533 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.851423 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.851398 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-2250b4-openshift-default-dockercfg-qrvc5\"" Nov 02 20:27:11.900075 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.900047 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb"] Nov 02 20:27:11.935366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935335 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935372 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935399 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkknn\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-kube-api-access-nkknn\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935423 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935446 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935476 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935821 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935530 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935821 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935571 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935917 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935835 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.935981 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935964 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.936036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.935983 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.936036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.936000 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.936264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.936245 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.936330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.936268 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.937737 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.937714 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.937897 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.937879 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.995183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.995152 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:11.995183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:11.995179 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkknn\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-kube-api-access-nkknn\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:12.116224 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.116155 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:12.294808 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.294771 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7"] Nov 02 20:27:12.298740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.298721 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:12.339747 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.339716 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733a96e9-08bd-48d8-a015-aac03b32ffd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7\" (UID: \"733a96e9-08bd-48d8-a015-aac03b32ffd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:12.339896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.339880 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdcn\" (UniqueName: \"kubernetes.io/projected/733a96e9-08bd-48d8-a015-aac03b32ffd6-kube-api-access-6vdcn\") pod \"payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7\" (UID: \"733a96e9-08bd-48d8-a015-aac03b32ffd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:12.350097 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.350071 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-30ee89\"" Nov 02 20:27:12.350237 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.350146 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-30ee89-dockercfg-qqj5d\"" Nov 02 20:27:12.352945 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.352913 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7"] Nov 02 20:27:12.441174 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.441106 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdcn\" (UniqueName: \"kubernetes.io/projected/733a96e9-08bd-48d8-a015-aac03b32ffd6-kube-api-access-6vdcn\") pod \"payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7\" (UID: \"733a96e9-08bd-48d8-a015-aac03b32ffd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:12.441174 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.441167 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733a96e9-08bd-48d8-a015-aac03b32ffd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7\" (UID: \"733a96e9-08bd-48d8-a015-aac03b32ffd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:12.441801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.441785 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733a96e9-08bd-48d8-a015-aac03b32ffd6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7\" (UID: \"733a96e9-08bd-48d8-a015-aac03b32ffd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:12.500079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.500050 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdcn\" (UniqueName: \"kubernetes.io/projected/733a96e9-08bd-48d8-a015-aac03b32ffd6-kube-api-access-6vdcn\") pod \"payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7\" (UID: \"733a96e9-08bd-48d8-a015-aac03b32ffd6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:12.530693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.530659 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb"] Nov 02 20:27:12.532325 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:12.532301 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bdecaa_9919_4802_b8f0_aaa24b2d0270.slice/crio-4e79fcf5d4accdf678475b45e01ac3e2fed8bdc9e8290f563543690913b9eed6 WatchSource:0}: Error finding container 4e79fcf5d4accdf678475b45e01ac3e2fed8bdc9e8290f563543690913b9eed6: Status 404 returned error can't find the container with id 4e79fcf5d4accdf678475b45e01ac3e2fed8bdc9e8290f563543690913b9eed6 Nov 02 20:27:12.534537 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.534512 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:12.534595 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.534571 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:12.534653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.534596 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:12.611985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.611956 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:12.879407 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.879377 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq"] Nov 02 20:27:12.884043 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.884023 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:12.932761 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.932731 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-a5cad4-serving-cert\"" Nov 02 20:27:12.932923 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.932811 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 20:27:12.932923 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.932864 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 20:27:12.933038 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.932958 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 20:27:12.933038 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.932985 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bg8z4\"" Nov 02 20:27:12.940454 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.938778 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq"] Nov 02 20:27:12.946512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.946487 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlbjr\" (UniqueName: \"kubernetes.io/projected/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-kube-api-access-zlbjr\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:12.946632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.946534 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-metrics-certs\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:12.946749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.946730 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:12.946807 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.946767 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:12.987172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:12.987155 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7"] Nov 02 20:27:12.989100 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:12.989075 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733a96e9_08bd_48d8_a015_aac03b32ffd6.slice/crio-194850f85c185d14d00f540035921c431b2fe4d86bc4c977ecca6a11c784b8a1 WatchSource:0}: Error finding container 194850f85c185d14d00f540035921c431b2fe4d86bc4c977ecca6a11c784b8a1: Status 404 returned error can't find the container with id 194850f85c185d14d00f540035921c431b2fe4d86bc4c977ecca6a11c784b8a1 Nov 02 20:27:13.047760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.047734 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlbjr\" (UniqueName: \"kubernetes.io/projected/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-kube-api-access-zlbjr\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.047880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.047777 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-metrics-certs\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.047938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.047882 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.047938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.047914 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.048055 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:27:13.048040 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-serving-cert: secret "maas-api-e2e-worker-w6-a5cad4-serving-cert" not found Nov 02 20:27:13.048145 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:27:13.048134 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls podName:cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f nodeName:}" failed. No retries permitted until 2026-11-02 20:27:13.548094039 +0000 UTC m=+705.912705236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls") pod "maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" (UID: "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f") : secret "maas-api-e2e-worker-w6-a5cad4-serving-cert" not found Nov 02 20:27:13.048629 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.048608 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.050099 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.050076 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-metrics-certs\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.105110 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.105081 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlbjr\" (UniqueName: \"kubernetes.io/projected/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-kube-api-access-zlbjr\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.320841 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.320809 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" event={"ID":"b2bdecaa-9919-4802-b8f0-aaa24b2d0270","Type":"ContainerStarted","Data":"91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb"} Nov 02 20:27:13.320994 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.320862 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" event={"ID":"b2bdecaa-9919-4802-b8f0-aaa24b2d0270","Type":"ContainerStarted","Data":"4e79fcf5d4accdf678475b45e01ac3e2fed8bdc9e8290f563543690913b9eed6"} Nov 02 20:27:13.322378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.322354 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" event={"ID":"733a96e9-08bd-48d8-a015-aac03b32ffd6","Type":"ContainerStarted","Data":"d3c43470cda3b4879063950db952d63d6123a8e5a0f3c5bfe27f7c8ac03c4fb3"} Nov 02 20:27:13.322557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.322402 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" event={"ID":"733a96e9-08bd-48d8-a015-aac03b32ffd6","Type":"ContainerStarted","Data":"194850f85c185d14d00f540035921c431b2fe4d86bc4c977ecca6a11c784b8a1"} Nov 02 20:27:13.322557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.322430 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:13.434904 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.434860 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" podStartSLOduration=2.434846762 podStartE2EDuration="2.434846762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:13.432583066 +0000 UTC m=+705.797194270" watchObservedRunningTime="2026-11-02 20:27:13.434846762 +0000 UTC m=+705.799457966" Nov 02 20:27:13.544099 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.544057 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" podStartSLOduration=1.544044486 podStartE2EDuration="1.544044486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:13.541915199 +0000 UTC m=+705.906526404" watchObservedRunningTime="2026-11-02 20:27:13.544044486 +0000 UTC m=+705.908655690" Nov 02 20:27:13.558952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.558916 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.561454 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.561431 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:13.798960 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:13.798724 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:14.116647 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:14.116570 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:14.122114 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:14.122088 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:14.211702 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:14.211089 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc6e351_3134_45b1_88cc_ae8cfd1f9c8f.slice/crio-24b3a5033bf8839e543258a130817a9859aeacf3ce0458aee428ee1787b823d2 WatchSource:0}: Error finding container 24b3a5033bf8839e543258a130817a9859aeacf3ce0458aee428ee1787b823d2: Status 404 returned error can't find the container with id 24b3a5033bf8839e543258a130817a9859aeacf3ce0458aee428ee1787b823d2 Nov 02 20:27:14.220285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:14.220259 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq"] Nov 02 20:27:14.329447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:14.329410 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" event={"ID":"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f","Type":"ContainerStarted","Data":"24b3a5033bf8839e543258a130817a9859aeacf3ce0458aee428ee1787b823d2"} Nov 02 20:27:14.329777 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:14.329758 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:14.330779 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:14.330759 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:27:15.336083 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:15.336042 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" event={"ID":"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f","Type":"ContainerStarted","Data":"5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4"} Nov 02 20:27:15.336493 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:15.336218 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:15.446148 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:15.446093 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" podStartSLOduration=2.637164685 podStartE2EDuration="3.446077193s" totalImagesPullingTime="808.912508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:15.445279214 +0000 UTC m=+707.809890420" watchObservedRunningTime="2026-11-02 20:27:15.446077193 +0000 UTC m=+707.810688400" Nov 02 20:27:18.144972 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.144945 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m"] Nov 02 20:27:18.148939 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.148918 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.197454 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.197425 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-bce768-serving-cert\"" Nov 02 20:27:18.212809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.212778 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.212953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.212827 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9sf\" (UniqueName: \"kubernetes.io/projected/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-kube-api-access-ct9sf\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.212953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.212896 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.213087 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.212997 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-metrics-certs\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.245872 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.245846 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m"] Nov 02 20:27:18.313645 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.313616 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.313816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.313702 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9sf\" (UniqueName: \"kubernetes.io/projected/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-kube-api-access-ct9sf\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.313816 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:27:18.313757 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-serving-cert: secret "maas-api-e2e-worker-w4-bce768-serving-cert" not found Nov 02 20:27:18.313946 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:27:18.313824 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls podName:13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979 nodeName:}" failed. No retries permitted until 2026-11-02 20:27:18.813802619 +0000 UTC m=+711.178413815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls") pod "maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" (UID: "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979") : secret "maas-api-e2e-worker-w4-bce768-serving-cert" not found Nov 02 20:27:18.313946 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.313763 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.313946 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.313893 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-metrics-certs\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.314397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.314372 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.316076 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.316056 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-metrics-certs\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.372953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.372927 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9sf\" (UniqueName: \"kubernetes.io/projected/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-kube-api-access-ct9sf\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.818367 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.818337 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:18.820714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:18.820687 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls\") pod \"maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:19.061321 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:19.061289 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:19.426160 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:19.426009 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m"] Nov 02 20:27:19.428752 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:19.428718 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d7b8cd_4e46_4c8e_bcdf_9e3ffaa9e979.slice/crio-b0d930e4970f3a70e620f9be568ba59d16e393e44649e7750d18cca1fcd663a2 WatchSource:0}: Error finding container b0d930e4970f3a70e620f9be568ba59d16e393e44649e7750d18cca1fcd663a2: Status 404 returned error can't find the container with id b0d930e4970f3a70e620f9be568ba59d16e393e44649e7750d18cca1fcd663a2 Nov 02 20:27:20.374181 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:20.374150 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" event={"ID":"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979","Type":"ContainerStarted","Data":"b0d930e4970f3a70e620f9be568ba59d16e393e44649e7750d18cca1fcd663a2"} Nov 02 20:27:21.345894 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:21.345867 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:27:21.384765 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:21.384724 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" event={"ID":"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979","Type":"ContainerStarted","Data":"004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8"} Nov 02 20:27:21.385368 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:21.385344 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:21.613347 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:21.613257 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" podStartSLOduration=2.6973659960000003 podStartE2EDuration="3.613243945s" totalImagesPullingTime="915.877949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:21.612076928 +0000 UTC m=+713.976688131" watchObservedRunningTime="2026-11-02 20:27:21.613243945 +0000 UTC m=+713.977855150" Nov 02 20:27:22.286279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.286250 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5"] Nov 02 20:27:22.290162 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.290141 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:22.338635 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.338609 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-61cdc1\"" Nov 02 20:27:22.338988 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.338969 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-61cdc1-dockercfg-jfmzf\"" Nov 02 20:27:22.351495 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.351461 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqst8\" (UniqueName: \"kubernetes.io/projected/f495bc83-6739-4f1c-9aae-cd8e54042154-kube-api-access-zqst8\") pod \"payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5\" (UID: \"f495bc83-6739-4f1c-9aae-cd8e54042154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:22.351898 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.351657 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f495bc83-6739-4f1c-9aae-cd8e54042154-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5\" (UID: \"f495bc83-6739-4f1c-9aae-cd8e54042154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:22.408664 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.408540 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5"] Nov 02 20:27:22.433130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.433092 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx"] Nov 02 20:27:22.437432 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.437412 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:22.453011 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.452990 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f495bc83-6739-4f1c-9aae-cd8e54042154-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5\" (UID: \"f495bc83-6739-4f1c-9aae-cd8e54042154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:22.453118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.453071 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqst8\" (UniqueName: \"kubernetes.io/projected/f495bc83-6739-4f1c-9aae-cd8e54042154-kube-api-access-zqst8\") pod \"payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5\" (UID: \"f495bc83-6739-4f1c-9aae-cd8e54042154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:22.453591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.453570 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f495bc83-6739-4f1c-9aae-cd8e54042154-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5\" (UID: \"f495bc83-6739-4f1c-9aae-cd8e54042154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:22.490991 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.490965 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx"] Nov 02 20:27:22.510065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.510035 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqst8\" (UniqueName: \"kubernetes.io/projected/f495bc83-6739-4f1c-9aae-cd8e54042154-kube-api-access-zqst8\") pod \"payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5\" (UID: \"f495bc83-6739-4f1c-9aae-cd8e54042154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:22.553964 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.553937 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/943db2e6-08cb-49d8-a8d9-f6abd8754b13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx\" (UID: \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:22.554097 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.554001 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sw4\" (UniqueName: \"kubernetes.io/projected/943db2e6-08cb-49d8-a8d9-f6abd8754b13-kube-api-access-l4sw4\") pod \"payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx\" (UID: \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:22.602792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.602769 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:22.660334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.654846 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/943db2e6-08cb-49d8-a8d9-f6abd8754b13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx\" (UID: \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:22.664906 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.664879 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sw4\" (UniqueName: \"kubernetes.io/projected/943db2e6-08cb-49d8-a8d9-f6abd8754b13-kube-api-access-l4sw4\") pod \"payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx\" (UID: \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:22.665186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.663515 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/943db2e6-08cb-49d8-a8d9-f6abd8754b13-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx\" (UID: \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:22.721944 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.721913 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sw4\" (UniqueName: \"kubernetes.io/projected/943db2e6-08cb-49d8-a8d9-f6abd8754b13-kube-api-access-l4sw4\") pod \"payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx\" (UID: \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:22.749776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.749752 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:22.974511 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:22.974485 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5"] Nov 02 20:27:22.976249 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:22.976222 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf495bc83_6739_4f1c_9aae_cd8e54042154.slice/crio-7b8f82546acdcc1dafe5f3c590ba7a06435ee40b1aa2c0cc8abc8db9f2bc25f9 WatchSource:0}: Error finding container 7b8f82546acdcc1dafe5f3c590ba7a06435ee40b1aa2c0cc8abc8db9f2bc25f9: Status 404 returned error can't find the container with id 7b8f82546acdcc1dafe5f3c590ba7a06435ee40b1aa2c0cc8abc8db9f2bc25f9 Nov 02 20:27:23.119295 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.119273 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx"] Nov 02 20:27:23.120936 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:23.120913 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943db2e6_08cb_49d8_a8d9_f6abd8754b13.slice/crio-f49b904d1fae68f5500a55fbb2e264fa09b68d660282b53676f0e4c64f0aa5a6 WatchSource:0}: Error finding container f49b904d1fae68f5500a55fbb2e264fa09b68d660282b53676f0e4c64f0aa5a6: Status 404 returned error can't find the container with id f49b904d1fae68f5500a55fbb2e264fa09b68d660282b53676f0e4c64f0aa5a6 Nov 02 20:27:23.419693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.419599 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" event={"ID":"943db2e6-08cb-49d8-a8d9-f6abd8754b13","Type":"ContainerStarted","Data":"4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6"} Nov 02 20:27:23.419693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.419651 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:23.419693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.419684 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" event={"ID":"943db2e6-08cb-49d8-a8d9-f6abd8754b13","Type":"ContainerStarted","Data":"f49b904d1fae68f5500a55fbb2e264fa09b68d660282b53676f0e4c64f0aa5a6"} Nov 02 20:27:23.421225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.421202 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" event={"ID":"f495bc83-6739-4f1c-9aae-cd8e54042154","Type":"ContainerStarted","Data":"646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba"} Nov 02 20:27:23.421347 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.421243 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" event={"ID":"f495bc83-6739-4f1c-9aae-cd8e54042154","Type":"ContainerStarted","Data":"7b8f82546acdcc1dafe5f3c590ba7a06435ee40b1aa2c0cc8abc8db9f2bc25f9"} Nov 02 20:27:23.421347 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.421282 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:23.528582 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.528536 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" podStartSLOduration=1.528524808 podStartE2EDuration="1.528524808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:23.526852216 +0000 UTC m=+715.891463421" watchObservedRunningTime="2026-11-02 20:27:23.528524808 +0000 UTC m=+715.893136012" Nov 02 20:27:23.634293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:23.634231 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" podStartSLOduration=1.6342187369999999 podStartE2EDuration="1.634218737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:23.633750199 +0000 UTC m=+715.998361405" watchObservedRunningTime="2026-11-02 20:27:23.634218737 +0000 UTC m=+715.998829980" Nov 02 20:27:24.332649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:24.332622 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:27:28.426425 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:28.426395 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:27:32.438929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.438891 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2"] Nov 02 20:27:32.444230 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.444208 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:32.497986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.497957 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2"] Nov 02 20:27:32.540613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.540589 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-694cee\"" Nov 02 20:27:32.540743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.540629 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-694cee-dockercfg-dwfzr\"" Nov 02 20:27:32.567551 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.567525 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj698\" (UniqueName: \"kubernetes.io/projected/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-kube-api-access-lj698\") pod \"payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2\" (UID: \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:32.567661 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.567588 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2\" (UID: \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:32.658379 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:27:32.658348 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:27:32.669176 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.669156 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj698\" (UniqueName: \"kubernetes.io/projected/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-kube-api-access-lj698\") pod \"payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2\" (UID: \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:32.669284 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.669218 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2\" (UID: \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:32.669886 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.669870 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2\" (UID: \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:32.726472 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.726413 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj698\" (UniqueName: \"kubernetes.io/projected/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-kube-api-access-lj698\") pod \"payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2\" (UID: \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:32.760810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:32.760783 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:33.135786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:33.135759 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2"] Nov 02 20:27:33.137882 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:27:33.137854 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2ce8ee_1763_4542_8c70_7cf8a2a4cc6b.slice/crio-bd9e75efe8a8a994c047da766bffbf29a5efcb76616c3f90f4f6526f0b35a665 WatchSource:0}: Error finding container bd9e75efe8a8a994c047da766bffbf29a5efcb76616c3f90f4f6526f0b35a665: Status 404 returned error can't find the container with id bd9e75efe8a8a994c047da766bffbf29a5efcb76616c3f90f4f6526f0b35a665 Nov 02 20:27:33.506505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:33.506420 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" event={"ID":"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b","Type":"ContainerStarted","Data":"26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c"} Nov 02 20:27:33.506505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:33.506483 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" event={"ID":"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b","Type":"ContainerStarted","Data":"bd9e75efe8a8a994c047da766bffbf29a5efcb76616c3f90f4f6526f0b35a665"} Nov 02 20:27:33.507073 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:33.506515 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:33.615292 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:33.615228 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" podStartSLOduration=1.615211098 podStartE2EDuration="1.615211098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:33.614459385 +0000 UTC m=+725.979070592" watchObservedRunningTime="2026-11-02 20:27:33.615211098 +0000 UTC m=+725.979822304" Nov 02 20:27:34.265018 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.264992 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h"] Nov 02 20:27:34.268855 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.268836 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.317386 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.317365 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-30ee89-serving-cert\"" Nov 02 20:27:34.324294 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.324261 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h"] Nov 02 20:27:34.388882 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.388822 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf33d81f-c923-417e-a5d7-48217518b55c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.389030 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.388886 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-metrics-certs\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.389030 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.388979 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2s2\" (UniqueName: \"kubernetes.io/projected/bf33d81f-c923-417e-a5d7-48217518b55c-kube-api-access-jl2s2\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.389030 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.389011 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.429868 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.429845 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:27:34.430006 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.429993 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:27:34.490546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.490493 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf33d81f-c923-417e-a5d7-48217518b55c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.490648 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.490563 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-metrics-certs\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.490648 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.490635 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2s2\" (UniqueName: \"kubernetes.io/projected/bf33d81f-c923-417e-a5d7-48217518b55c-kube-api-access-jl2s2\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.490803 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.490663 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.490862 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:27:34.490842 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-serving-cert: secret "maas-api-e2e-worker-w0-30ee89-serving-cert" not found Nov 02 20:27:34.490920 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:27:34.490898 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls podName:bf33d81f-c923-417e-a5d7-48217518b55c nodeName:}" failed. No retries permitted until 2026-11-02 20:27:34.990881009 +0000 UTC m=+727.355492193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls") pod "maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" (UID: "bf33d81f-c923-417e-a5d7-48217518b55c") : secret "maas-api-e2e-worker-w0-30ee89-serving-cert" not found Nov 02 20:27:34.491200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.491179 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf33d81f-c923-417e-a5d7-48217518b55c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.493014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.492993 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-metrics-certs\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:34.519337 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.519302 5420 generic.go:356] "Generic (PLEG): container finished" podID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerID="d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1" exitCode=0 Nov 02 20:27:34.519754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.519349 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" event={"ID":"fc9eb207-3df4-4ba5-acda-4041ef5bc080","Type":"ContainerDied","Data":"d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1"} Nov 02 20:27:34.519933 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.519900 5420 scope.go:122] "RemoveContainer" containerID="d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1" Nov 02 20:27:34.521503 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.521466 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:34.521718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.521557 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:34.521718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.521600 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:34.551111 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:34.551088 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2s2\" (UniqueName: \"kubernetes.io/projected/bf33d81f-c923-417e-a5d7-48217518b55c-kube-api-access-jl2s2\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:35.008442 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.008409 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:35.011546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.011506 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:35.185695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.185085 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:35.317710 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.315181 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:35.535129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.535100 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" event={"ID":"fc9eb207-3df4-4ba5-acda-4041ef5bc080","Type":"ContainerStarted","Data":"24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631"} Nov 02 20:27:35.601072 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.601031 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h"] Nov 02 20:27:35.968743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.967020 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:35.974699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.973772 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:27:35.974699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:35.973829 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:27:36.543033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:36.543000 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" event={"ID":"bf33d81f-c923-417e-a5d7-48217518b55c","Type":"ContainerStarted","Data":"8cba87ebb5b816022ec56b22e3d9de9cdbefde2bb2c03d54c4b4f65727d222b5"} Nov 02 20:27:36.973858 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:36.973825 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:27:36.974149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:36.974111 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:27:37.549827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:37.549784 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" event={"ID":"bf33d81f-c923-417e-a5d7-48217518b55c","Type":"ContainerStarted","Data":"edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e"} Nov 02 20:27:37.550214 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:37.549890 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:37.691840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:37.691726 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" podStartSLOduration=2.814127692 podStartE2EDuration="3.691702892s" totalImagesPullingTime="877.5752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:27:37.683193825 +0000 UTC m=+730.047805029" watchObservedRunningTime="2026-11-02 20:27:37.691702892 +0000 UTC m=+730.056314092" Nov 02 20:27:37.972147 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:37.972062 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:27:37.972400 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:37.972369 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:27:38.966098 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:38.966061 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:27:38.966724 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:38.966655 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:27:39.965152 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:39.965113 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 02 20:27:39.965427 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:39.965397 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 02 20:27:40.969770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:40.969743 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:40.970197 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:40.970104 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:40.970880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:40.970856 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:27:43.562014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:43.561978 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:27:43.610311 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:43.610286 5420 generic.go:356] "Generic (PLEG): container finished" podID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerID="deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889" exitCode=0 Nov 02 20:27:43.610450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:43.610337 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" event={"ID":"a5b79a65-9be4-4acb-b68b-19dca7de84cd","Type":"ContainerDied","Data":"deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889"} Nov 02 20:27:43.610659 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:43.610645 5420 scope.go:122] "RemoveContainer" containerID="deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889" Nov 02 20:27:43.612017 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:43.611989 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:43.612127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:43.612054 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:43.612127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:43.612085 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:44.523031 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:44.523002 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:27:44.624102 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:44.624051 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" event={"ID":"a5b79a65-9be4-4acb-b68b-19dca7de84cd","Type":"ContainerStarted","Data":"4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2"} Nov 02 20:27:45.296696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:45.296050 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:45.296696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:45.296176 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:27:45.296696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:45.296220 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:27:46.293448 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:46.293003 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:27:46.293448 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:46.293072 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:27:47.296521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:47.296067 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:27:47.296521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:47.296131 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:27:48.305693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:48.294763 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:27:48.305693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:48.294822 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:27:49.293002 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:49.292966 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 02 20:27:49.293272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:49.293245 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 02 20:27:50.279696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:50.278776 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:50.297203 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:50.297175 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:50.661779 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:50.661751 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:27:54.689453 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:54.689423 5420 generic.go:356] "Generic (PLEG): container finished" podID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerID="892f37a058392d43b648e78c0ed91a358c46eb9ad26bca11132f5990a2fb029e" exitCode=0 Nov 02 20:27:54.689908 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:54.689495 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" event={"ID":"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3","Type":"ContainerDied","Data":"892f37a058392d43b648e78c0ed91a358c46eb9ad26bca11132f5990a2fb029e"} Nov 02 20:27:54.689908 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:54.689875 5420 scope.go:122] "RemoveContainer" containerID="892f37a058392d43b648e78c0ed91a358c46eb9ad26bca11132f5990a2fb029e" Nov 02 20:27:54.691039 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:54.691012 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:54.691110 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:54.691084 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:54.691149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:54.691118 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:27:55.698934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:55.698889 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" event={"ID":"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3","Type":"ContainerStarted","Data":"efc9df80052f5a624e93d7f54b8abee6aaa4bee100c4bec2cb56875ba395c889"} Nov 02 20:27:56.382039 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:56.382006 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:27:56.382406 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:56.382311 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 20:27:56.382582 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:56.382560 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 20:27:57.382996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:57.382959 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 20:27:57.383385 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:57.383027 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 20:27:58.385372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:58.384780 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 20:27:58.385372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:58.384853 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 20:27:59.382695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:59.382103 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 20:27:59.382695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:27:59.382177 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 20:28:00.382834 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:00.382801 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 02 20:28:00.383222 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:00.382855 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 02 20:28:01.386483 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:01.386452 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:28:01.386926 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:01.386885 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:28:01.387552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:01.387530 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:28:02.658355 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:28:02.658322 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:28:11.818923 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:11.818895 5420 generic.go:356] "Generic (PLEG): container finished" podID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerID="91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb" exitCode=0 Nov 02 20:28:11.819330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:11.818965 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" event={"ID":"b2bdecaa-9919-4802-b8f0-aaa24b2d0270","Type":"ContainerDied","Data":"91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb"} Nov 02 20:28:11.819330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:11.819258 5420 scope.go:122] "RemoveContainer" containerID="91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb" Nov 02 20:28:11.820445 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:11.820420 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:11.820511 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:11.820487 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:11.820545 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:11.820515 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:28:12.826745 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:12.826715 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" event={"ID":"b2bdecaa-9919-4802-b8f0-aaa24b2d0270","Type":"ContainerStarted","Data":"792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a"} Nov 02 20:28:13.116514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:13.116431 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:28:13.117497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:13.117471 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:28:13.117616 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:13.117522 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:28:14.117214 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:14.117182 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:28:14.117697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:14.117243 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:28:14.330501 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:14.330470 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:28:15.116741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:15.116708 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:28:15.116958 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:15.116756 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:28:16.117051 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:16.117017 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:28:16.117441 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:16.117093 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:28:17.117477 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:17.117442 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 20:28:17.117927 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:17.117507 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 20:28:18.120635 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:18.120605 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:28:18.121495 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:18.121478 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:28:32.657378 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:28:32.657333 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:28:56.736317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.736281 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn"] Nov 02 20:28:56.740500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.740481 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.809616 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.809586 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-model-cache\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.809616 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.809616 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tls-certs\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.809859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.809698 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-home\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.809859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.809732 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4m26\" (UniqueName: \"kubernetes.io/projected/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-kube-api-access-f4m26\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.809859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.809802 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-dshm\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.809859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.809822 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tmp-dir\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.835108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.835077 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a5cad4\"/\"kube-root-ca.crt\"" Nov 02 20:28:56.835215 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.835120 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a5cad4\"/\"openshift-service-ca.crt\"" Nov 02 20:28:56.835215 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.835140 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a5cad4\"/\"premium-w6-a5cad4-kserve-self-signed-certs\"" Nov 02 20:28:56.835215 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.835191 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a5cad4\"/\"default-dockercfg-j46wd\"" Nov 02 20:28:56.837533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.837515 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn"] Nov 02 20:28:56.911037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911006 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-home\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.911217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911046 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4m26\" (UniqueName: \"kubernetes.io/projected/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-kube-api-access-f4m26\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.911217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911094 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-dshm\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.911217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911113 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tmp-dir\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.911217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911188 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-model-cache\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.911217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911204 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tls-certs\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.911526 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911393 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-home\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.911583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911569 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tmp-dir\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.911632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.911612 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-model-cache\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.913360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.913337 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-dshm\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.913740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.913722 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tls-certs\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:56.968402 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:56.968370 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4m26\" (UniqueName: \"kubernetes.io/projected/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-kube-api-access-f4m26\") pod \"premium-w6-a5cad4-kserve-5889c85cc8-sk7wn\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:57.052187 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:57.052163 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:28:57.434640 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:57.434607 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn"] Nov 02 20:28:57.435838 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:28:57.435800 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b6bdb14_0bc7_4c61_856c_38f9ec36a9a6.slice/crio-c60c70e203166b8cfecbf85269a435bd92478fa04cdfe43d02b6d3eefe2b1a5c WatchSource:0}: Error finding container c60c70e203166b8cfecbf85269a435bd92478fa04cdfe43d02b6d3eefe2b1a5c: Status 404 returned error can't find the container with id c60c70e203166b8cfecbf85269a435bd92478fa04cdfe43d02b6d3eefe2b1a5c Nov 02 20:28:58.087370 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:58.087334 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" event={"ID":"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6","Type":"ContainerStarted","Data":"91702de1c8da01f920b0324ebaf4359bca285f9bab6a121756add26a185707b3"} Nov 02 20:28:58.087370 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:58.087376 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" event={"ID":"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6","Type":"ContainerStarted","Data":"c60c70e203166b8cfecbf85269a435bd92478fa04cdfe43d02b6d3eefe2b1a5c"} Nov 02 20:28:58.196173 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:58.196120 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podStartSLOduration=2.196108229 podStartE2EDuration="2.196108229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:28:58.195638623 +0000 UTC m=+810.560249830" watchObservedRunningTime="2026-11-02 20:28:58.196108229 +0000 UTC m=+810.560719433" Nov 02 20:28:59.052485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:28:59.052460 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:29:02.659642 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:29:02.659602 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:29:07.053324 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:07.053292 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:29:07.066023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:07.065994 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:29:07.158978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:07.158948 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:29:32.658698 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:29:32.658644 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:29:32.816632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.816604 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb"] Nov 02 20:29:32.820714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.820695 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:32.915557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.915502 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"default-dockercfg-jb4t8\"" Nov 02 20:29:32.915702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.915557 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"kube-root-ca.crt\"" Nov 02 20:29:32.915702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.915638 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"openshift-service-ca.crt\"" Nov 02 20:29:32.915702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.915641 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-bce768\"/\"premium-w4-bce768-kserve-self-signed-certs\"" Nov 02 20:29:32.918338 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.918316 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb"] Nov 02 20:29:32.959209 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.959174 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-dshm\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:32.960811 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.960786 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-tmp-dir\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:32.963645 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.963623 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-model-cache\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:32.963980 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.963950 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f812f95a-615e-49c3-8362-bcca78368fbc-tls-certs\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:32.964098 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.963987 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf8xj\" (UniqueName: \"kubernetes.io/projected/f812f95a-615e-49c3-8362-bcca78368fbc-kube-api-access-bf8xj\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:32.964183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:32.964096 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-home\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.064839 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.064811 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f812f95a-615e-49c3-8362-bcca78368fbc-tls-certs\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.064985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.064850 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf8xj\" (UniqueName: \"kubernetes.io/projected/f812f95a-615e-49c3-8362-bcca78368fbc-kube-api-access-bf8xj\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.064985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.064909 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-home\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.064985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.064948 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-dshm\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.064985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.064968 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-tmp-dir\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.065212 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.064997 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-model-cache\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.065429 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.065383 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-tmp-dir\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.065429 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.065406 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-model-cache\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.065575 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.065481 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-home\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.068123 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.068102 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-dshm\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.068289 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.068270 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f812f95a-615e-49c3-8362-bcca78368fbc-tls-certs\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.121423 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.121398 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf8xj\" (UniqueName: \"kubernetes.io/projected/f812f95a-615e-49c3-8362-bcca78368fbc-kube-api-access-bf8xj\") pod \"premium-w4-bce768-kserve-6dcccf759c-px8lb\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.132258 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.132238 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:33.497135 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:33.497113 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb"] Nov 02 20:29:33.499010 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:29:33.498973 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf812f95a_615e_49c3_8362_bcca78368fbc.slice/crio-78eacd345de0ea99ee21d957b95c5e4f744b454ed2e3b164c1a120772a3729b8 WatchSource:0}: Error finding container 78eacd345de0ea99ee21d957b95c5e4f744b454ed2e3b164c1a120772a3729b8: Status 404 returned error can't find the container with id 78eacd345de0ea99ee21d957b95c5e4f744b454ed2e3b164c1a120772a3729b8 Nov 02 20:29:34.300910 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:34.300869 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" event={"ID":"f812f95a-615e-49c3-8362-bcca78368fbc","Type":"ContainerStarted","Data":"b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb"} Nov 02 20:29:34.301256 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:34.300920 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" event={"ID":"f812f95a-615e-49c3-8362-bcca78368fbc","Type":"ContainerStarted","Data":"78eacd345de0ea99ee21d957b95c5e4f744b454ed2e3b164c1a120772a3729b8"} Nov 02 20:29:34.412112 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:34.412058 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podStartSLOduration=2.412045338 podStartE2EDuration="2.412045338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:29:34.409515629 +0000 UTC m=+846.774126833" watchObservedRunningTime="2026-11-02 20:29:34.412045338 +0000 UTC m=+846.776656543" Nov 02 20:29:35.132931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:35.132898 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:43.132532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:43.132500 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:43.145131 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:43.145108 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:43.366896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:43.366866 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:29:59.877166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:59.877133 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n"] Nov 02 20:29:59.881033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:59.881014 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:29:59.933073 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:59.933046 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c49693\"/\"kube-root-ca.crt\"" Nov 02 20:29:59.975953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:59.975927 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c49693\"/\"default-dockercfg-ndc7b\"" Nov 02 20:29:59.975953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:59.975941 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-c49693\"/\"openshift-service-ca.crt\"" Nov 02 20:29:59.976117 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:59.975972 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c49693\"/\"premium-w5-c49693-kserve-self-signed-certs\"" Nov 02 20:29:59.978605 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:29:59.978587 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n"] Nov 02 20:30:00.024779 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.024750 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-tmp-dir\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.024955 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.024789 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-model-cache\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.024955 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.024894 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f9583e-31b6-4da0-b01f-5eace8a27f16-tls-certs\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.025144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.024951 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-dshm\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.025144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.025071 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r59x5\" (UniqueName: \"kubernetes.io/projected/36f9583e-31b6-4da0-b01f-5eace8a27f16-kube-api-access-r59x5\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.025144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.025105 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-home\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126296 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126263 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-dshm\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126309 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r59x5\" (UniqueName: \"kubernetes.io/projected/36f9583e-31b6-4da0-b01f-5eace8a27f16-kube-api-access-r59x5\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126328 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-home\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126357 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-tmp-dir\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126384 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-model-cache\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126437 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f9583e-31b6-4da0-b01f-5eace8a27f16-tls-certs\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126753 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-home\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126917 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126873 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-tmp-dir\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.126987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.126933 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-model-cache\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.128655 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.128605 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-dshm\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.128888 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.128873 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f9583e-31b6-4da0-b01f-5eace8a27f16-tls-certs\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.177281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.177255 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86"] Nov 02 20:30:00.181455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.181436 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:00.189563 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.189544 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r59x5\" (UniqueName: \"kubernetes.io/projected/36f9583e-31b6-4da0-b01f-5eace8a27f16-kube-api-access-r59x5\") pod \"premium-w5-c49693-kserve-dfb67579c-7fm9n\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.192925 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.192902 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:00.195264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.195244 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8"] Nov 02 20:30:00.199427 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.199407 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:00.241482 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.238877 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86"] Nov 02 20:30:00.281793 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.281768 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8"] Nov 02 20:30:00.329084 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.329054 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:30:00.341663 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.341628 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcgdx\" (UniqueName: \"kubernetes.io/projected/eff7e7d2-91b9-4a47-9880-3293518bc460-kube-api-access-bcgdx\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8\" (UID: \"eff7e7d2-91b9-4a47-9880-3293518bc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:00.341817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.341698 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plgvb\" (UniqueName: \"kubernetes.io/projected/15490557-075a-483b-9d70-f7ac0c140b35-kube-api-access-plgvb\") pod \"maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86\" (UID: \"15490557-075a-483b-9d70-f7ac0c140b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:00.341817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.341795 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15490557-075a-483b-9d70-f7ac0c140b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86\" (UID: \"15490557-075a-483b-9d70-f7ac0c140b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:00.341968 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.341944 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff7e7d2-91b9-4a47-9880-3293518bc460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8\" (UID: \"eff7e7d2-91b9-4a47-9880-3293518bc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:00.442459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.442396 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15490557-075a-483b-9d70-f7ac0c140b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86\" (UID: \"15490557-075a-483b-9d70-f7ac0c140b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:00.442459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.442459 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff7e7d2-91b9-4a47-9880-3293518bc460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8\" (UID: \"eff7e7d2-91b9-4a47-9880-3293518bc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:00.442643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.442507 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcgdx\" (UniqueName: \"kubernetes.io/projected/eff7e7d2-91b9-4a47-9880-3293518bc460-kube-api-access-bcgdx\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8\" (UID: \"eff7e7d2-91b9-4a47-9880-3293518bc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:00.442643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.442528 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plgvb\" (UniqueName: \"kubernetes.io/projected/15490557-075a-483b-9d70-f7ac0c140b35-kube-api-access-plgvb\") pod \"maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86\" (UID: \"15490557-075a-483b-9d70-f7ac0c140b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:00.443130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.443107 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15490557-075a-483b-9d70-f7ac0c140b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86\" (UID: \"15490557-075a-483b-9d70-f7ac0c140b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:00.443273 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.443253 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff7e7d2-91b9-4a47-9880-3293518bc460-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8\" (UID: \"eff7e7d2-91b9-4a47-9880-3293518bc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:00.500094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.500067 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcgdx\" (UniqueName: \"kubernetes.io/projected/eff7e7d2-91b9-4a47-9880-3293518bc460-kube-api-access-bcgdx\") pod \"maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8\" (UID: \"eff7e7d2-91b9-4a47-9880-3293518bc460\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:00.500192 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.500067 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plgvb\" (UniqueName: \"kubernetes.io/projected/15490557-075a-483b-9d70-f7ac0c140b35-kube-api-access-plgvb\") pod \"maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86\" (UID: \"15490557-075a-483b-9d70-f7ac0c140b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:00.503070 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.503054 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:00.579718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.579661 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:00.583825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.583731 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n"] Nov 02 20:30:00.587576 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:30:00.587547 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36f9583e_31b6_4da0_b01f_5eace8a27f16.slice/crio-e4ce7e84ae0a2d5ca768a211a33d845d0af7d9a1364d10c59abe2e3dcd66f566 WatchSource:0}: Error finding container e4ce7e84ae0a2d5ca768a211a33d845d0af7d9a1364d10c59abe2e3dcd66f566: Status 404 returned error can't find the container with id e4ce7e84ae0a2d5ca768a211a33d845d0af7d9a1364d10c59abe2e3dcd66f566 Nov 02 20:30:00.869303 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.869279 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86"] Nov 02 20:30:00.871371 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:30:00.871334 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15490557_075a_483b_9d70_f7ac0c140b35.slice/crio-f1a4a8e0ef9bfb5a42499b62ff0d01951116128d93b40a55b15d3f9a1a55e02e WatchSource:0}: Error finding container f1a4a8e0ef9bfb5a42499b62ff0d01951116128d93b40a55b15d3f9a1a55e02e: Status 404 returned error can't find the container with id f1a4a8e0ef9bfb5a42499b62ff0d01951116128d93b40a55b15d3f9a1a55e02e Nov 02 20:30:00.951627 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:00.951556 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8"] Nov 02 20:30:00.957882 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:30:00.957857 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff7e7d2_91b9_4a47_9880_3293518bc460.slice/crio-ed4a669792eaba72cc1932ea06a97202908abe503d902c2352019ddd234f9a07 WatchSource:0}: Error finding container ed4a669792eaba72cc1932ea06a97202908abe503d902c2352019ddd234f9a07: Status 404 returned error can't find the container with id ed4a669792eaba72cc1932ea06a97202908abe503d902c2352019ddd234f9a07 Nov 02 20:30:01.464551 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:01.464522 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" event={"ID":"15490557-075a-483b-9d70-f7ac0c140b35","Type":"ContainerStarted","Data":"f1a4a8e0ef9bfb5a42499b62ff0d01951116128d93b40a55b15d3f9a1a55e02e"} Nov 02 20:30:01.465930 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:01.465900 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" event={"ID":"eff7e7d2-91b9-4a47-9880-3293518bc460","Type":"ContainerStarted","Data":"ed4a669792eaba72cc1932ea06a97202908abe503d902c2352019ddd234f9a07"} Nov 02 20:30:01.467440 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:01.467419 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" event={"ID":"36f9583e-31b6-4da0-b01f-5eace8a27f16","Type":"ContainerStarted","Data":"5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad"} Nov 02 20:30:01.467573 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:01.467445 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" event={"ID":"36f9583e-31b6-4da0-b01f-5eace8a27f16","Type":"ContainerStarted","Data":"e4ce7e84ae0a2d5ca768a211a33d845d0af7d9a1364d10c59abe2e3dcd66f566"} Nov 02 20:30:01.578721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:01.578661 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podStartSLOduration=2.578648783 podStartE2EDuration="2.578648783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:30:01.576721522 +0000 UTC m=+873.941332728" watchObservedRunningTime="2026-11-02 20:30:01.578648783 +0000 UTC m=+873.943259987" Nov 02 20:30:02.193755 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:02.193723 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:02.658827 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:02.658793 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:30:06.506258 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:06.506175 5420 generic.go:356] "Generic (PLEG): container finished" podID="15490557-075a-483b-9d70-f7ac0c140b35" containerID="536b87de1129d91d5fedbda6fabffc87c11b4cd0ed343a5d9973d92c665f0d97" exitCode=60 Nov 02 20:30:06.506783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:06.506261 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" event={"ID":"15490557-075a-483b-9d70-f7ac0c140b35","Type":"ContainerDied","Data":"536b87de1129d91d5fedbda6fabffc87c11b4cd0ed343a5d9973d92c665f0d97"} Nov 02 20:30:06.506783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:06.506479 5420 scope.go:122] "RemoveContainer" containerID="536b87de1129d91d5fedbda6fabffc87c11b4cd0ed343a5d9973d92c665f0d97" Nov 02 20:30:06.508311 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:06.508284 5420 generic.go:356] "Generic (PLEG): container finished" podID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerID="1d0fb6e7c565c40457231b6a27528f6473ed4a34122d1a7455ae97989654a155" exitCode=60 Nov 02 20:30:06.508418 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:06.508339 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" event={"ID":"eff7e7d2-91b9-4a47-9880-3293518bc460","Type":"ContainerDied","Data":"1d0fb6e7c565c40457231b6a27528f6473ed4a34122d1a7455ae97989654a155"} Nov 02 20:30:06.508535 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:06.508518 5420 scope.go:122] "RemoveContainer" containerID="1d0fb6e7c565c40457231b6a27528f6473ed4a34122d1a7455ae97989654a155" Nov 02 20:30:07.516842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:07.516819 5420 generic.go:356] "Generic (PLEG): container finished" podID="15490557-075a-483b-9d70-f7ac0c140b35" containerID="ed17740e79034cfcdf3e23a9ec3ced0687dbee3fd106a703e2d6403eae094db8" exitCode=60 Nov 02 20:30:07.517211 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:07.516905 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" event={"ID":"15490557-075a-483b-9d70-f7ac0c140b35","Type":"ContainerDied","Data":"ed17740e79034cfcdf3e23a9ec3ced0687dbee3fd106a703e2d6403eae094db8"} Nov 02 20:30:07.517211 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:07.516965 5420 scope.go:122] "RemoveContainer" containerID="536b87de1129d91d5fedbda6fabffc87c11b4cd0ed343a5d9973d92c665f0d97" Nov 02 20:30:07.517211 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:07.517192 5420 scope.go:122] "RemoveContainer" containerID="ed17740e79034cfcdf3e23a9ec3ced0687dbee3fd106a703e2d6403eae094db8" Nov 02 20:30:07.517505 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:07.517448 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86_odh-ai-gateway-infra(15490557-075a-483b-9d70-f7ac0c140b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" podUID="15490557-075a-483b-9d70-f7ac0c140b35" Nov 02 20:30:07.518847 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:07.518829 5420 generic.go:356] "Generic (PLEG): container finished" podID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerID="d187eee6e3110114083909df183a7f208545df275d2f36b9e3e431b904563726" exitCode=60 Nov 02 20:30:07.518947 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:07.518886 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" event={"ID":"eff7e7d2-91b9-4a47-9880-3293518bc460","Type":"ContainerDied","Data":"d187eee6e3110114083909df183a7f208545df275d2f36b9e3e431b904563726"} Nov 02 20:30:07.519130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:07.519115 5420 scope.go:122] "RemoveContainer" containerID="d187eee6e3110114083909df183a7f208545df275d2f36b9e3e431b904563726" Nov 02 20:30:07.519338 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:07.519319 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8_odh-ai-gateway-infra(eff7e7d2-91b9-4a47-9880-3293518bc460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" Nov 02 20:30:07.534581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:07.534562 5420 scope.go:122] "RemoveContainer" containerID="1d0fb6e7c565c40457231b6a27528f6473ed4a34122d1a7455ae97989654a155" Nov 02 20:30:08.526257 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:08.526231 5420 scope.go:122] "RemoveContainer" containerID="d187eee6e3110114083909df183a7f208545df275d2f36b9e3e431b904563726" Nov 02 20:30:08.526721 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:08.526456 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8_odh-ai-gateway-infra(eff7e7d2-91b9-4a47-9880-3293518bc460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" Nov 02 20:30:08.527822 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:08.527804 5420 scope.go:122] "RemoveContainer" containerID="ed17740e79034cfcdf3e23a9ec3ced0687dbee3fd106a703e2d6403eae094db8" Nov 02 20:30:08.527973 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:08.527952 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86_odh-ai-gateway-infra(15490557-075a-483b-9d70-f7ac0c140b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" podUID="15490557-075a-483b-9d70-f7ac0c140b35" Nov 02 20:30:10.193128 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:10.193099 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:10.204872 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:10.204844 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:10.551709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:10.551662 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:30:18.767191 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:18.767163 5420 scope.go:122] "RemoveContainer" containerID="ed17740e79034cfcdf3e23a9ec3ced0687dbee3fd106a703e2d6403eae094db8" Nov 02 20:30:19.596475 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:19.596448 5420 generic.go:356] "Generic (PLEG): container finished" podID="15490557-075a-483b-9d70-f7ac0c140b35" containerID="8ba761cca7491f458809e648fa32bee2c1e15b2f10a3c6e0772e1ea7c21d2062" exitCode=60 Nov 02 20:30:19.596648 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:19.596523 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" event={"ID":"15490557-075a-483b-9d70-f7ac0c140b35","Type":"ContainerDied","Data":"8ba761cca7491f458809e648fa32bee2c1e15b2f10a3c6e0772e1ea7c21d2062"} Nov 02 20:30:19.596648 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:19.596582 5420 scope.go:122] "RemoveContainer" containerID="ed17740e79034cfcdf3e23a9ec3ced0687dbee3fd106a703e2d6403eae094db8" Nov 02 20:30:19.596911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:19.596896 5420 scope.go:122] "RemoveContainer" containerID="8ba761cca7491f458809e648fa32bee2c1e15b2f10a3c6e0772e1ea7c21d2062" Nov 02 20:30:19.597170 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:19.597152 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86_odh-ai-gateway-infra(15490557-075a-483b-9d70-f7ac0c140b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" podUID="15490557-075a-483b-9d70-f7ac0c140b35" Nov 02 20:30:19.767089 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:19.767033 5420 scope.go:122] "RemoveContainer" containerID="d187eee6e3110114083909df183a7f208545df275d2f36b9e3e431b904563726" Nov 02 20:30:20.516334 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:20.516300 5420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff7e7d2_91b9_4a47_9880_3293518bc460.slice/crio-conmon-79602b90352436e74fbfb91772b60a5aa9c84c5c91531403306f72790e389229.scope\": RecentStats: unable to find data in memory cache]" Nov 02 20:30:20.604478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.604448 5420 generic.go:356] "Generic (PLEG): container finished" podID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerID="79602b90352436e74fbfb91772b60a5aa9c84c5c91531403306f72790e389229" exitCode=60 Nov 02 20:30:20.604684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.604526 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" event={"ID":"eff7e7d2-91b9-4a47-9880-3293518bc460","Type":"ContainerDied","Data":"79602b90352436e74fbfb91772b60a5aa9c84c5c91531403306f72790e389229"} Nov 02 20:30:20.604684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.604586 5420 scope.go:122] "RemoveContainer" containerID="d187eee6e3110114083909df183a7f208545df275d2f36b9e3e431b904563726" Nov 02 20:30:20.604991 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.604969 5420 scope.go:122] "RemoveContainer" containerID="79602b90352436e74fbfb91772b60a5aa9c84c5c91531403306f72790e389229" Nov 02 20:30:20.605285 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:20.605253 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8_odh-ai-gateway-infra(eff7e7d2-91b9-4a47-9880-3293518bc460)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" Nov 02 20:30:20.714109 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.714050 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86"] Nov 02 20:30:20.851920 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.851896 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:20.942816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.942791 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15490557-075a-483b-9d70-f7ac0c140b35-kube-api-access-plgvb\" (UniqueName: \"kubernetes.io/projected/15490557-075a-483b-9d70-f7ac0c140b35-kube-api-access-plgvb\") pod \"15490557-075a-483b-9d70-f7ac0c140b35\" (UID: \"15490557-075a-483b-9d70-f7ac0c140b35\") " Nov 02 20:30:20.942952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.942877 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15490557-075a-483b-9d70-f7ac0c140b35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15490557-075a-483b-9d70-f7ac0c140b35-maas-api-service-ca\") pod \"15490557-075a-483b-9d70-f7ac0c140b35\" (UID: \"15490557-075a-483b-9d70-f7ac0c140b35\") " Nov 02 20:30:20.943200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.943181 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15490557-075a-483b-9d70-f7ac0c140b35-maas-api-service-ca" pod "15490557-075a-483b-9d70-f7ac0c140b35" (UID: "15490557-075a-483b-9d70-f7ac0c140b35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:20.944662 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:20.944641 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15490557-075a-483b-9d70-f7ac0c140b35-kube-api-access-plgvb" pod "15490557-075a-483b-9d70-f7ac0c140b35" (UID: "15490557-075a-483b-9d70-f7ac0c140b35"). InnerVolumeSpecName "kube-api-access-plgvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:21.043872 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.043803 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plgvb\" (UniqueName: \"kubernetes.io/projected/15490557-075a-483b-9d70-f7ac0c140b35-kube-api-access-plgvb\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.043872 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.043845 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15490557-075a-483b-9d70-f7ac0c140b35-maas-api-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:21.614607 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.614577 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" event={"ID":"15490557-075a-483b-9d70-f7ac0c140b35","Type":"ContainerDied","Data":"f1a4a8e0ef9bfb5a42499b62ff0d01951116128d93b40a55b15d3f9a1a55e02e"} Nov 02 20:30:21.614607 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.614611 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86" Nov 02 20:30:21.615098 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.614631 5420 scope.go:122] "RemoveContainer" containerID="8ba761cca7491f458809e648fa32bee2c1e15b2f10a3c6e0772e1ea7c21d2062" Nov 02 20:30:21.726794 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.726770 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8"] Nov 02 20:30:21.776419 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.776391 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86"] Nov 02 20:30:21.776419 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.776425 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-694cee-29894190-q9h86"] Nov 02 20:30:21.858707 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.858683 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:21.952733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.952621 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff7e7d2-91b9-4a47-9880-3293518bc460-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff7e7d2-91b9-4a47-9880-3293518bc460-maas-api-service-ca\") pod \"eff7e7d2-91b9-4a47-9880-3293518bc460\" (UID: \"eff7e7d2-91b9-4a47-9880-3293518bc460\") " Nov 02 20:30:21.952917 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.952828 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff7e7d2-91b9-4a47-9880-3293518bc460-kube-api-access-bcgdx\" (UniqueName: \"kubernetes.io/projected/eff7e7d2-91b9-4a47-9880-3293518bc460-kube-api-access-bcgdx\") pod \"eff7e7d2-91b9-4a47-9880-3293518bc460\" (UID: \"eff7e7d2-91b9-4a47-9880-3293518bc460\") " Nov 02 20:30:21.956645 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.956347 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff7e7d2-91b9-4a47-9880-3293518bc460-maas-api-service-ca" pod "eff7e7d2-91b9-4a47-9880-3293518bc460" (UID: "eff7e7d2-91b9-4a47-9880-3293518bc460"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:30:21.958803 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:21.958762 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff7e7d2-91b9-4a47-9880-3293518bc460-kube-api-access-bcgdx" pod "eff7e7d2-91b9-4a47-9880-3293518bc460" (UID: "eff7e7d2-91b9-4a47-9880-3293518bc460"). InnerVolumeSpecName "kube-api-access-bcgdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:30:22.054533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:22.054505 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcgdx\" (UniqueName: \"kubernetes.io/projected/eff7e7d2-91b9-4a47-9880-3293518bc460-kube-api-access-bcgdx\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:22.054533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:22.054529 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eff7e7d2-91b9-4a47-9880-3293518bc460-maas-api-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:30:22.621690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:22.621649 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" Nov 02 20:30:22.622093 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:22.621696 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8" event={"ID":"eff7e7d2-91b9-4a47-9880-3293518bc460","Type":"ContainerDied","Data":"ed4a669792eaba72cc1932ea06a97202908abe503d902c2352019ddd234f9a07"} Nov 02 20:30:22.622093 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:22.621796 5420 scope.go:122] "RemoveContainer" containerID="79602b90352436e74fbfb91772b60a5aa9c84c5c91531403306f72790e389229" Nov 02 20:30:22.776433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:22.776389 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8"] Nov 02 20:30:22.781858 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:22.781167 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-bce768-29894190-hm4v8"] Nov 02 20:30:23.773571 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:23.773536 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15490557-075a-483b-9d70-f7ac0c140b35" path="/var/lib/kubelet/pods/15490557-075a-483b-9d70-f7ac0c140b35/volumes" Nov 02 20:30:23.773969 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:30:23.773911 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" path="/var/lib/kubelet/pods/eff7e7d2-91b9-4a47-9880-3293518bc460/volumes" Nov 02 20:30:32.660689 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:30:32.660627 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:31:02.658572 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:31:02.658488 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:31:19.965508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.965476 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn"] Nov 02 20:31:19.966033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.965969 5420 cpu_manager.go:414] "containerMap: removing container" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:19.966033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.965981 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:19.966033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966023 5420 cpu_manager.go:414] "containerMap: removing container" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.966033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966029 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.966166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966050 5420 cpu_manager.go:414] "containerMap: removing container" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:19.966166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966059 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:19.966166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966072 5420 cpu_manager.go:414] "containerMap: removing container" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.966166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966080 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.966166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966089 5420 cpu_manager.go:414] "containerMap: removing container" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.966166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966094 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.966359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966212 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:19.966359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966230 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.966359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966243 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:19.966359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966252 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.966359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.966262 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff7e7d2-91b9-4a47-9880-3293518bc460" containerName="cleanup" Nov 02 20:31:19.969722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:19.969703 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.017986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.017962 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.018101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018011 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.018101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018083 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.018187 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018156 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.018234 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018189 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.018234 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018215 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.018302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018278 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-694cee-openshift-default-dockercfg-tv8jc\"" Nov 02 20:31:20.018348 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018325 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.018388 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018367 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8mq\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-kube-api-access-4c8mq\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.018426 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.018396 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.068733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.068710 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn"] Nov 02 20:31:20.119404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119367 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.119547 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119423 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8mq\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-kube-api-access-4c8mq\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.119547 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119451 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.119547 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119490 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.119547 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119537 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.119793 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119583 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.119793 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119695 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.119793 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119743 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.119793 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.119771 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.120145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.120036 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.120145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.120039 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.120145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.120078 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.120301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.120268 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.120569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.120542 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.121936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.121915 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.122145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.122125 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.176353 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.176327 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.176613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.176593 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8mq\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-kube-api-access-4c8mq\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.283649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.283558 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:20.707591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.707567 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn"] Nov 02 20:31:20.710019 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:31:20.709982 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fd9aed_bf60_4b23_a5f1_c166b2c48f2b.slice/crio-fd680d7c1a2c185c7b32d4caeedd0e1df1bea59eae843adde4e4c95027e4e5ff WatchSource:0}: Error finding container fd680d7c1a2c185c7b32d4caeedd0e1df1bea59eae843adde4e4c95027e4e5ff: Status 404 returned error can't find the container with id fd680d7c1a2c185c7b32d4caeedd0e1df1bea59eae843adde4e4c95027e4e5ff Nov 02 20:31:20.712467 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.712440 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:20.712544 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.712499 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:20.712544 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.712528 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:20.992161 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.992078 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" event={"ID":"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b","Type":"ContainerStarted","Data":"e7fbd18432301be0b036cf894c072148f9e85093f33ab36437f98417f68ad712"} Nov 02 20:31:20.992161 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:20.992137 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" event={"ID":"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b","Type":"ContainerStarted","Data":"fd680d7c1a2c185c7b32d4caeedd0e1df1bea59eae843adde4e4c95027e4e5ff"} Nov 02 20:31:21.107115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:21.107064 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podStartSLOduration=2.107049618 podStartE2EDuration="2.107049618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:21.105348302 +0000 UTC m=+953.469959501" watchObservedRunningTime="2026-11-02 20:31:21.107049618 +0000 UTC m=+953.471660821" Nov 02 20:31:21.284875 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:21.284765 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:21.285878 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:21.285851 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:31:21.286019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:21.285928 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:31:22.284027 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:22.283991 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:31:22.284582 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:22.284555 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:31:23.287032 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:23.286996 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:31:23.287599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:23.287566 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:31:24.287380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:24.287343 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:31:24.287986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:24.287957 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:31:25.284490 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:25.284050 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:31:25.284490 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:25.284113 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:31:26.284700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:26.284046 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Nov 02 20:31:26.284700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:26.284120 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Nov 02 20:31:27.370031 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:27.369998 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:27.370462 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:27.370341 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:27.371036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:27.371011 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:31:32.657821 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:31:32.657782 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:31:34.190581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.190536 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v"] Nov 02 20:31:34.191349 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.191330 5420 cpu_manager.go:414] "containerMap: removing container" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:34.191393 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.191353 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:34.191506 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.191493 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="15490557-075a-483b-9d70-f7ac0c140b35" containerName="cleanup" Nov 02 20:31:34.195573 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.195550 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.292894 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.292860 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a5cad4-openshift-default-dockercfg-phnwd\"" Nov 02 20:31:34.296949 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.296920 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v"] Nov 02 20:31:34.299968 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.299941 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.300149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.300119 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.300271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.300193 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.300271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.300262 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfbmv\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-kube-api-access-xfbmv\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.300388 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.300329 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.300388 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.300381 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52f9ae8-24ab-4006-b444-9e3971071196-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.300497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.300414 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52f9ae8-24ab-4006-b444-9e3971071196-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.300497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.300452 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.300605 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.300500 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.401865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.401827 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.401865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.401870 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402005 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402111 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfbmv\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-kube-api-access-xfbmv\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402239 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402189 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402245 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52f9ae8-24ab-4006-b444-9e3971071196-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402278 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52f9ae8-24ab-4006-b444-9e3971071196-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402390 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402320 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402390 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402340 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402390 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402377 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402529 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402383 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402710 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402659 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.402887 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.402790 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.403113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.403076 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52f9ae8-24ab-4006-b444-9e3971071196-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.404585 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.404563 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.404815 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.404797 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52f9ae8-24ab-4006-b444-9e3971071196-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.459870 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.459808 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.460101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.460081 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfbmv\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-kube-api-access-xfbmv\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.510875 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.510849 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:34.932386 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.932327 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v"] Nov 02 20:31:34.933733 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:31:34.933708 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52f9ae8_24ab_4006_b444_9e3971071196.slice/crio-22ebb6736948c601ad460776da09da9cae0848f8b4191b822a449a4a72bf5ded WatchSource:0}: Error finding container 22ebb6736948c601ad460776da09da9cae0848f8b4191b822a449a4a72bf5ded: Status 404 returned error can't find the container with id 22ebb6736948c601ad460776da09da9cae0848f8b4191b822a449a4a72bf5ded Nov 02 20:31:34.935906 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.935873 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:34.935987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.935935 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:34.935987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:34.935963 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:31:35.080864 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:35.080830 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" event={"ID":"a52f9ae8-24ab-4006-b444-9e3971071196","Type":"ContainerStarted","Data":"31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b"} Nov 02 20:31:35.081036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:35.080886 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" event={"ID":"a52f9ae8-24ab-4006-b444-9e3971071196","Type":"ContainerStarted","Data":"22ebb6736948c601ad460776da09da9cae0848f8b4191b822a449a4a72bf5ded"} Nov 02 20:31:35.195211 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:35.195096 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podStartSLOduration=1.195083911 podStartE2EDuration="1.195083911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:31:35.192094106 +0000 UTC m=+967.556705310" watchObservedRunningTime="2026-11-02 20:31:35.195083911 +0000 UTC m=+967.559695126" Nov 02 20:31:35.512094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:35.512013 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:35.513411 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:35.513387 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:31:35.513523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:35.513444 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:31:36.511711 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:36.511663 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:31:36.512075 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:36.511737 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:31:37.511688 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:37.511634 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:31:37.511849 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:37.511723 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:31:38.348791 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:38.348757 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh"] Nov 02 20:31:38.348969 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:38.348947 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" containerID="cri-o://0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7" gracePeriod=30 Nov 02 20:31:38.511487 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:38.511457 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:31:38.511643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:38.511511 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:31:39.511406 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:39.511375 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:31:39.511826 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:39.511422 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:31:40.511804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:40.511774 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 20:31:40.512152 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:40.511821 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 20:31:41.515920 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:41.515891 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:41.516320 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:41.516283 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:41.516974 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:41.516954 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:31:43.604337 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.604314 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:31:43.720573 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720503 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-credential-socket\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720737 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720570 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91fab4a-d096-46f5-b4b4-6e912edc7785-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b91fab4a-d096-46f5-b4b4-6e912edc7785-istiod-ca-cert\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720737 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720604 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-podinfo\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720737 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720625 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-envoy\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720737 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720665 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-data\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720975 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720789 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-socket\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720975 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720808 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-credential-socket" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.720975 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720845 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-certs\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720975 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720875 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-token\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-token\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720975 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720946 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-kube-api-access-7cjxn\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-kube-api-access-7cjxn\") pod \"b91fab4a-d096-46f5-b4b4-6e912edc7785\" (UID: \"b91fab4a-d096-46f5-b4b4-6e912edc7785\") " Nov 02 20:31:43.720975 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.720952 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-data" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.721291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.721085 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-socket" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.721291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.721107 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91fab4a-d096-46f5-b4b4-6e912edc7785-istiod-ca-cert" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:31:43.721291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.721203 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-certs" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.721457 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.721363 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.721457 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.721385 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.721457 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.721398 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b91fab4a-d096-46f5-b4b4-6e912edc7785-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.721457 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.721412 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.721457 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.721424 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.723029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.723004 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-envoy" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:31:43.723155 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.723134 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-podinfo" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:31:43.723231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.723147 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-token" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:43.723231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.723196 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-kube-api-access-7cjxn" pod "b91fab4a-d096-46f5-b4b4-6e912edc7785" (UID: "b91fab4a-d096-46f5-b4b4-6e912edc7785"). InnerVolumeSpecName "kube-api-access-7cjxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:31:43.822192 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.822165 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjxn\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-kube-api-access-7cjxn\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.822192 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.822194 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.822360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.822209 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:43.822360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:43.822221 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b91fab4a-d096-46f5-b4b4-6e912edc7785-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:31:44.141828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.141784 5420 generic.go:356] "Generic (PLEG): container finished" podID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerID="0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7" exitCode=0 Nov 02 20:31:44.142015 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.141823 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" event={"ID":"b91fab4a-d096-46f5-b4b4-6e912edc7785","Type":"ContainerDied","Data":"0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7"} Nov 02 20:31:44.142015 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.141854 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" Nov 02 20:31:44.142015 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.141888 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh" event={"ID":"b91fab4a-d096-46f5-b4b4-6e912edc7785","Type":"ContainerDied","Data":"ab62a8d97eaf650e39e325dd8074da88ae400050e3444d4442d2fb931f5b731d"} Nov 02 20:31:44.142015 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.141918 5420 scope.go:122] "RemoveContainer" containerID="0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7" Nov 02 20:31:44.154368 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.154338 5420 scope.go:122] "RemoveContainer" containerID="0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7" Nov 02 20:31:44.154648 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:31:44.154630 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7\": container with ID starting with 0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7 not found: ID does not exist" containerID="0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7" Nov 02 20:31:44.154743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.154657 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7"} err="failed to get container status \"0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7\": rpc error: code = NotFound desc = could not find container \"0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7\": container with ID starting with 0147148270cb9c2a067d185e460210b60688d5705a8879d0e9be5815657230c7 not found: ID does not exist" Nov 02 20:31:44.295185 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.295155 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh"] Nov 02 20:31:44.341095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:44.341069 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-swdjh"] Nov 02 20:31:45.774963 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:45.774935 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" path="/var/lib/kubelet/pods/b91fab4a-d096-46f5-b4b4-6e912edc7785/volumes" Nov 02 20:31:54.451939 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:54.451894 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8vqkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:31:54.452571 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:31:54.451951 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8vqkm" podUID="eccd3c31-d213-4763-b79b-ee970b2c334d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:32:02.658745 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:32:02.658703 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:32:05.623545 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.623511 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7"] Nov 02 20:32:05.624101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.624085 5420 cpu_manager.go:414] "containerMap: removing container" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" Nov 02 20:32:05.624101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.624101 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" Nov 02 20:32:05.624221 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.624191 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91fab4a-d096-46f5-b4b4-6e912edc7785" containerName="istio-proxy" Nov 02 20:32:05.629959 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.629939 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.678708 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.678640 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-c49693-openshift-default-dockercfg-cq675\"" Nov 02 20:32:05.711105 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711068 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.711105 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711113 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.711345 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711259 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.711345 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711321 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.711450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711361 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8accdb-1605-45c0-9402-8299d48e0b31-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.711450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711418 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.711553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711457 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.711553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711476 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9zqk\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-kube-api-access-h9zqk\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.711694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.711646 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.727367 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.727335 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7"] Nov 02 20:32:05.812248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812212 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812244 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812285 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812310 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812343 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8accdb-1605-45c0-9402-8299d48e0b31-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812377 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812759 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812734 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812791 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812830 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zqk\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-kube-api-access-h9zqk\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812945 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812866 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.812945 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812890 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.813045 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812787 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.813045 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.812976 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.813153 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.813084 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8accdb-1605-45c0-9402-8299d48e0b31-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.814775 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.814748 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.814987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.814971 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.869774 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.869749 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.870093 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.870073 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zqk\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-kube-api-access-h9zqk\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:05.943291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:05.943228 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:06.362025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:06.361992 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7"] Nov 02 20:32:06.369352 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:06.369312 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:06.369454 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:06.369399 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:06.369454 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:06.369438 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:07.330302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:07.330269 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" event={"ID":"cd8accdb-1605-45c0-9402-8299d48e0b31","Type":"ContainerStarted","Data":"e437a7789a2a95509ce0c368b2cdffdc0cc580132913a43acec91e104d80c6e6"} Nov 02 20:32:07.330770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:07.330323 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" event={"ID":"cd8accdb-1605-45c0-9402-8299d48e0b31","Type":"ContainerStarted","Data":"55b84849d28722edc6d876305bf3349f336ddc594cc10c9c4ea0e11d0987a193"} Nov 02 20:32:07.493879 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:07.493822 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podStartSLOduration=2.49380862 podStartE2EDuration="2.49380862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:07.447331141 +0000 UTC m=+999.811942346" watchObservedRunningTime="2026-11-02 20:32:07.49380862 +0000 UTC m=+999.858419872" Nov 02 20:32:07.943660 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:07.943624 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:07.945223 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:07.945199 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:32:07.945352 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:07.945254 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:32:08.392176 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.392148 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6"] Nov 02 20:32:08.397497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.397477 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.497179 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.497148 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6"] Nov 02 20:32:08.544014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.543978 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.544170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.544021 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.544217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.544174 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.544260 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.544219 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.544311 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.544294 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.544366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.544352 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2tst\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-kube-api-access-c2tst\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.544436 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.544422 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.544473 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.544458 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.544512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.544497 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645065 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645102 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645138 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645200 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645225 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645277 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645301 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645318 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645362 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2tst\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-kube-api-access-c2tst\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645664 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645628 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645767 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645743 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645767 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645758 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.645866 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.645802 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.646057 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.646038 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.647905 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.647873 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.648192 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.648171 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.702761 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.702735 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2tst\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-kube-api-access-c2tst\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.703119 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.703095 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-85fc6\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.712841 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.712818 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:08.944360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.944273 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:32:08.944360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:08.944329 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:32:09.190170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.190143 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6"] Nov 02 20:32:09.192523 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:32:09.192491 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7017b523_af4a_4f4a_9f82_c9faeb58ff7c.slice/crio-1b5ac186dd63f49d1d5bb405a9f1a780b4ac18fde3188cb204fb76b430ce76db WatchSource:0}: Error finding container 1b5ac186dd63f49d1d5bb405a9f1a780b4ac18fde3188cb204fb76b430ce76db: Status 404 returned error can't find the container with id 1b5ac186dd63f49d1d5bb405a9f1a780b4ac18fde3188cb204fb76b430ce76db Nov 02 20:32:09.194976 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.194922 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:09.195050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.194981 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:09.195050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.195009 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:09.344942 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.344907 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" event={"ID":"7017b523-af4a-4f4a-9f82-c9faeb58ff7c","Type":"ContainerStarted","Data":"db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297"} Nov 02 20:32:09.345126 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.344960 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" event={"ID":"7017b523-af4a-4f4a-9f82-c9faeb58ff7c","Type":"ContainerStarted","Data":"1b5ac186dd63f49d1d5bb405a9f1a780b4ac18fde3188cb204fb76b430ce76db"} Nov 02 20:32:09.474746 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.464094 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podStartSLOduration=1.464063658 podStartE2EDuration="1.464063658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:09.457557827 +0000 UTC m=+1001.822169032" watchObservedRunningTime="2026-11-02 20:32:09.464063658 +0000 UTC m=+1001.828674863" Nov 02 20:32:09.713214 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.713185 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:09.944013 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.943979 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:32:09.944198 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:09.944041 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:32:10.720695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:10.720372 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:32:10.720695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:10.720447 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:32:10.946225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:10.946188 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:32:10.946505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:10.946460 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:32:11.744696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:11.740984 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:32:11.744696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:11.741043 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:32:11.946703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:11.946031 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:32:11.946703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:11.946091 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:32:12.933718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:12.933648 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:32:12.934278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:12.934247 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:32:12.946632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:12.946601 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 20:32:12.947093 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:12.947061 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 20:32:13.713371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:13.713335 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:32:13.713528 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:13.713399 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:32:14.713817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:14.713779 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:32:14.714226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:14.713840 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:32:14.803525 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:14.803495 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:14.804122 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:14.804104 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:14.805706 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:14.805659 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:32:15.713642 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:15.713607 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:32:15.713864 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:15.713685 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:32:16.713466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:16.713433 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 20:32:16.713636 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:16.713496 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 20:32:18.754816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:18.754775 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:32:18.755308 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:18.754824 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:32:19.755319 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:19.755285 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-85fc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:32:19.755728 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:19.755353 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:32:19.779599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:19.779554 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:19.780701 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:19.780659 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:19.781831 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:19.781806 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:32:23.389922 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.389888 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27"] Nov 02 20:32:23.394220 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.394187 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.466351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.466312 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.466689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.466647 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c3566-3158-4235-b01f-fce9d9205c69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.466856 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.466837 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.466977 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.466959 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.467129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.467113 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c3566-3158-4235-b01f-fce9d9205c69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.467248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.467232 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.467385 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.467369 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hv5\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-kube-api-access-69hv5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.467549 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.467530 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.468497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.468468 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.548998 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.548966 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27"] Nov 02 20:32:23.572445 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.572407 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.572770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.572748 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c3566-3158-4235-b01f-fce9d9205c69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.572903 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.572874 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.572992 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.572978 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.573100 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.573083 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.573237 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.573222 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c3566-3158-4235-b01f-fce9d9205c69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.573347 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.573334 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.573460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.573448 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hv5\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-kube-api-access-69hv5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.573588 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.573577 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.573742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.573729 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.574076 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.574048 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.575208 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.573454 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c3566-3158-4235-b01f-fce9d9205c69-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.575208 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.575190 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.575500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.575467 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.578992 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.578893 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.579099 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.578998 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c3566-3158-4235-b01f-fce9d9205c69-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.634997 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.634930 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.635848 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.635826 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hv5\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-kube-api-access-69hv5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-s9k27\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:23.710167 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:23.710068 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:24.481614 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:24.463096 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27"] Nov 02 20:32:24.606209 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:24.606171 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:24.606463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:24.606440 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:24.606632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:24.606612 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:24.612938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:24.612687 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" event={"ID":"606c3566-3158-4235-b01f-fce9d9205c69","Type":"ContainerStarted","Data":"e457f8bab857148d18e4d65668194c9b46f5c1ba38f1797a26ca05e6d0d012fe"} Nov 02 20:32:25.620137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:25.620099 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" event={"ID":"606c3566-3158-4235-b01f-fce9d9205c69","Type":"ContainerStarted","Data":"977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281"} Nov 02 20:32:25.710903 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:25.710871 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:25.712067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:25.712038 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s9k27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:32:25.712212 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:25.712105 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:32:25.813382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:25.813336 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podStartSLOduration=2.813323596 podStartE2EDuration="2.813323596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:25.813148296 +0000 UTC m=+1018.177759501" watchObservedRunningTime="2026-11-02 20:32:25.813323596 +0000 UTC m=+1018.177934801" Nov 02 20:32:26.710562 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:26.710532 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s9k27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:32:26.711060 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:26.710580 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:32:27.710978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:27.710943 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s9k27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:32:27.711381 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:27.711022 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:32:28.713034 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:28.712998 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s9k27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:32:28.713416 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:28.713089 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:32:29.711130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:29.711091 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s9k27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:32:29.711345 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:29.711144 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:32:30.712138 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:30.712103 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s9k27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:32:30.712659 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:30.712151 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:32:31.710327 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:31.710287 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-s9k27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 02 20:32:31.710521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:31.710336 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 02 20:32:32.662131 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:32:32.662095 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:32:32.938542 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:32.938461 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:32.938976 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:32.938930 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:32.939714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:32.939665 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:32:34.977192 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:34.977159 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj"] Nov 02 20:32:34.982325 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:34.982304 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.057871 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.057845 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.058001 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.057886 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.058062 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.058013 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.058114 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.058072 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.058158 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.058134 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.058192 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.058172 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.058258 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.058246 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d935d8-83fb-458c-a4ef-f0a0baf669be-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.058296 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.058266 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2vv\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-kube-api-access-4h2vv\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.058340 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.058295 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.082127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.082100 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj"] Nov 02 20:32:35.159836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.159808 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.159978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.159859 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.159978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.159892 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.159978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.159921 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.159978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.159971 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d935d8-83fb-458c-a4ef-f0a0baf669be-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.159998 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2vv\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-kube-api-access-4h2vv\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.160046 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.160119 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.160162 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.160250 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.160263 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.160444 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.160513 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.160743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.160722 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d935d8-83fb-458c-a4ef-f0a0baf669be-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.162450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.162430 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.162655 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.162639 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.220324 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.220287 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.220470 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.220432 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2vv\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-kube-api-access-4h2vv\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.298032 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.297966 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:35.721335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.721313 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj"] Nov 02 20:32:35.723537 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:32:35.723512 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d935d8_83fb_458c_a4ef_f0a0baf669be.slice/crio-2be8ac2ceae3cfc325025372cd53c68a784c717a3806d307b2f3ce53f8acc6e4 WatchSource:0}: Error finding container 2be8ac2ceae3cfc325025372cd53c68a784c717a3806d307b2f3ce53f8acc6e4: Status 404 returned error can't find the container with id 2be8ac2ceae3cfc325025372cd53c68a784c717a3806d307b2f3ce53f8acc6e4 Nov 02 20:32:35.725780 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.725752 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:35.725853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.725812 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:35.725853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:35.725839 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:36.695170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:36.695132 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" event={"ID":"54d935d8-83fb-458c-a4ef-f0a0baf669be","Type":"ContainerStarted","Data":"3c9af6f70ec9bef9bd8b528a82938230158e15b0eb8ee4f508356be71b01f9f1"} Nov 02 20:32:36.695697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:36.695189 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" event={"ID":"54d935d8-83fb-458c-a4ef-f0a0baf669be","Type":"ContainerStarted","Data":"2be8ac2ceae3cfc325025372cd53c68a784c717a3806d307b2f3ce53f8acc6e4"} Nov 02 20:32:36.901631 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:36.901579 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podStartSLOduration=2.901565772 podStartE2EDuration="2.901565772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:36.853344386 +0000 UTC m=+1029.217955591" watchObservedRunningTime="2026-11-02 20:32:36.901565772 +0000 UTC m=+1029.266176976" Nov 02 20:32:37.298749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:37.298714 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:37.300601 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:37.300580 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:32:37.300659 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:37.300640 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 20:32:38.298533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:38.298498 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:32:38.298937 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:38.298549 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 20:32:39.298964 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:39.298928 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:32:39.299410 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:39.298986 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 20:32:40.299290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:40.299255 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:32:40.299817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:40.299306 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 20:32:41.299375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:41.299345 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 20:32:41.299830 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:41.299623 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 20:32:42.439329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:42.439299 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:42.440294 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:42.440267 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:42.440854 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:42.440837 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:32:49.756911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.756880 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9"] Nov 02 20:32:49.761036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.761016 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.859429 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.859394 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9"] Nov 02 20:32:49.876809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.876782 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.876809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.876813 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvcc9\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-kube-api-access-zvcc9\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.877002 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.876965 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.877066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.877033 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.877225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.877163 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.877301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.877223 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.877301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.877243 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5974eac6-37e4-4cdd-bb61-b1946397079a-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.877301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.877279 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.877424 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.877384 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978530 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978505 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978551 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978570 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5974eac6-37e4-4cdd-bb61-b1946397079a-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978781 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978738 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978781 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978768 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978885 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978790 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978885 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978806 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvcc9\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-kube-api-access-zvcc9\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978898 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978929 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.978987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.978957 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.979271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.979247 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.979393 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.979346 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.979463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.979450 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.979634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.979614 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5974eac6-37e4-4cdd-bb61-b1946397079a-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.981090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.981072 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:49.981449 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:49.981429 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:50.036163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.036113 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvcc9\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-kube-api-access-zvcc9\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:50.036163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.036133 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:50.075849 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.075821 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:50.495463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.495441 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9"] Nov 02 20:32:50.496441 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:32:50.496416 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5974eac6_37e4_4cdd_bb61_b1946397079a.slice/crio-7e552b1694016e8adb98303191709282c3e872661a07331adba81d4bf9460742 WatchSource:0}: Error finding container 7e552b1694016e8adb98303191709282c3e872661a07331adba81d4bf9460742: Status 404 returned error can't find the container with id 7e552b1694016e8adb98303191709282c3e872661a07331adba81d4bf9460742 Nov 02 20:32:50.498666 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.498635 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:50.498775 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.498724 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:50.498775 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.498768 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:32:50.822072 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.822026 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" event={"ID":"5974eac6-37e4-4cdd-bb61-b1946397079a","Type":"ContainerStarted","Data":"656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5"} Nov 02 20:32:50.822537 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.822114 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" event={"ID":"5974eac6-37e4-4cdd-bb61-b1946397079a","Type":"ContainerStarted","Data":"7e552b1694016e8adb98303191709282c3e872661a07331adba81d4bf9460742"} Nov 02 20:32:50.937000 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:50.936932 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podStartSLOduration=1.936913796 podStartE2EDuration="1.936913796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:32:50.934766271 +0000 UTC m=+1043.299377474" watchObservedRunningTime="2026-11-02 20:32:50.936913796 +0000 UTC m=+1043.301525001" Nov 02 20:32:51.082798 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:51.081034 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:51.082798 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:51.082502 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:32:51.082798 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:51.082567 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:32:52.077694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:52.076967 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:32:52.077694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:52.077020 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:32:53.079695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:53.076969 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:32:53.079695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:53.077038 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:32:54.077072 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:54.077023 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:32:54.077395 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:54.077359 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:32:55.081707 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:55.079508 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:32:55.081707 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:55.079576 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:32:56.080513 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:56.080431 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 20:32:56.080513 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:56.080502 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 20:32:57.080466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:57.080438 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:57.080954 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:57.080843 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:32:57.081568 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:32:57.081550 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:33:02.658695 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:33:02.658648 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:33:08.132897 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.132863 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2"] Nov 02 20:33:08.138074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.138045 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.200858 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.200829 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2"] Nov 02 20:33:08.235332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.235297 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c49693\"/\"distinct-w5-c49693-kserve-self-signed-certs\"" Nov 02 20:33:08.279464 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.279429 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-home\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.279638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.279484 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx688\" (UniqueName: \"kubernetes.io/projected/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-kube-api-access-dx688\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.279638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.279545 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tls-certs\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.279638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.279603 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-dshm\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.279798 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.279752 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-model-cache\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.279798 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.279777 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tmp-dir\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381088 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381055 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-home\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381096 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx688\" (UniqueName: \"kubernetes.io/projected/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-kube-api-access-dx688\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381138 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tls-certs\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381177 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-dshm\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381445 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381324 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-model-cache\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381445 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381347 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tmp-dir\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381520 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-home\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381633 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-model-cache\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.381961 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.381929 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tmp-dir\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.384010 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.383952 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-dshm\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.384154 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.384139 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tls-certs\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.437761 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.437737 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx688\" (UniqueName: \"kubernetes.io/projected/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-kube-api-access-dx688\") pod \"distinct-w5-c49693-kserve-5d9fbb9556-fxmj2\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.458596 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.458573 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:08.878416 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.878391 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2"] Nov 02 20:33:08.880931 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:33:08.880898 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fcc7055_32d1_4eeb_a99f_1a8037dd89f8.slice/crio-e9319258dd0fe9dfe208fbb8291e27f411a57bf3b849bc5cf601ce62b399451c WatchSource:0}: Error finding container e9319258dd0fe9dfe208fbb8291e27f411a57bf3b849bc5cf601ce62b399451c: Status 404 returned error can't find the container with id e9319258dd0fe9dfe208fbb8291e27f411a57bf3b849bc5cf601ce62b399451c Nov 02 20:33:08.946899 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:08.946872 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" event={"ID":"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8","Type":"ContainerStarted","Data":"e9319258dd0fe9dfe208fbb8291e27f411a57bf3b849bc5cf601ce62b399451c"} Nov 02 20:33:09.954547 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:09.954514 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" event={"ID":"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8","Type":"ContainerStarted","Data":"008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04"} Nov 02 20:33:10.065260 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:10.065203 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podStartSLOduration=2.065186197 podStartE2EDuration="2.065186197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:10.064620455 +0000 UTC m=+1062.429231659" watchObservedRunningTime="2026-11-02 20:33:10.065186197 +0000 UTC m=+1062.429797404" Nov 02 20:33:10.458924 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:10.458886 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:14.445810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.445766 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm"] Nov 02 20:33:14.451185 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.451161 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.532264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.532163 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2250b4\"/\"openshift-service-ca.crt\"" Nov 02 20:33:14.556329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.556293 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-tmp-dir\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.556493 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.556355 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a0b642-a089-430d-8d23-008780082a49-tls-certs\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.556562 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.556531 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5sj9\" (UniqueName: \"kubernetes.io/projected/59a0b642-a089-430d-8d23-008780082a49-kube-api-access-k5sj9\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.556620 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.556571 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-model-cache\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.556712 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.556682 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-dshm\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.556792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.556775 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-home\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.578480 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.578448 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2250b4\"/\"default-dockercfg-swxfm\"" Nov 02 20:33:14.578713 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.578696 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2250b4\"/\"kube-root-ca.crt\"" Nov 02 20:33:14.580778 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.580603 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2250b4\"/\"distinct-w1-2250b4-kserve-self-signed-certs\"" Nov 02 20:33:14.588939 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.588917 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm"] Nov 02 20:33:14.657893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.657857 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5sj9\" (UniqueName: \"kubernetes.io/projected/59a0b642-a089-430d-8d23-008780082a49-kube-api-access-k5sj9\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.658115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.657909 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-model-cache\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.658115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.657961 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-dshm\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.658115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.658072 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-home\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.658295 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.658119 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-tmp-dir\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.658295 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.658178 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a0b642-a089-430d-8d23-008780082a49-tls-certs\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.659041 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.658857 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-model-cache\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.659186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.659142 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-home\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.659349 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.659323 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-tmp-dir\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.662913 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.662483 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a0b642-a089-430d-8d23-008780082a49-tls-certs\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.664090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.663323 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-dshm\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.725756 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.722969 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5sj9\" (UniqueName: \"kubernetes.io/projected/59a0b642-a089-430d-8d23-008780082a49-kube-api-access-k5sj9\") pod \"distinct-w1-2250b4-kserve-775ccc87d7-dg5fm\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:14.780374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:14.780062 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:15.255193 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:15.255170 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm"] Nov 02 20:33:15.256847 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:33:15.256811 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59a0b642_a089_430d_8d23_008780082a49.slice/crio-f118fef896c3e55147204d0aafeb83272a27303b2572a2e13602af59e3fb5a67 WatchSource:0}: Error finding container f118fef896c3e55147204d0aafeb83272a27303b2572a2e13602af59e3fb5a67: Status 404 returned error can't find the container with id f118fef896c3e55147204d0aafeb83272a27303b2572a2e13602af59e3fb5a67 Nov 02 20:33:16.020308 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:16.020278 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" event={"ID":"59a0b642-a089-430d-8d23-008780082a49","Type":"ContainerStarted","Data":"8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0"} Nov 02 20:33:16.020716 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:16.020319 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" event={"ID":"59a0b642-a089-430d-8d23-008780082a49","Type":"ContainerStarted","Data":"f118fef896c3e55147204d0aafeb83272a27303b2572a2e13602af59e3fb5a67"} Nov 02 20:33:16.165915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:16.165862 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podStartSLOduration=2.165845971 podStartE2EDuration="2.165845971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:16.165469449 +0000 UTC m=+1068.530080653" watchObservedRunningTime="2026-11-02 20:33:16.165845971 +0000 UTC m=+1068.530457176" Nov 02 20:33:16.780559 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:16.780519 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:18.459245 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:18.459210 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:18.471330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:18.471308 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:19.050235 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:19.050205 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:33:24.780749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:24.780715 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:24.793271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:24.793247 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:25.086376 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:25.086346 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:33:32.660585 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:33:32.660546 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:33:42.455638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.455563 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn"] Nov 02 20:33:42.461008 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.460989 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.490605 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.490580 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-model-cache\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.490765 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.490684 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-dshm\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.490765 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.490726 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e1d983-6558-405b-ad3a-58f3121815e6-tls-certs\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.490874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.490854 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-home\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.490944 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.490928 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-tmp-dir\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.491037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.491023 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdfb\" (UniqueName: \"kubernetes.io/projected/e9e1d983-6558-405b-ad3a-58f3121815e6-kube-api-access-zpdfb\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.556926 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.556901 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn"] Nov 02 20:33:42.558328 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.558304 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"openshift-service-ca.crt\"" Nov 02 20:33:42.558441 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.558331 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"kube-root-ca.crt\"" Nov 02 20:33:42.558494 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.558444 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"default-dockercfg-sdqsf\"" Nov 02 20:33:42.558548 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.558515 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-694cee\"/\"distinct-w2-694cee-kserve-self-signed-certs\"" Nov 02 20:33:42.591971 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.591947 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-model-cache\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.592101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.592000 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-dshm\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.592101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.592027 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e1d983-6558-405b-ad3a-58f3121815e6-tls-certs\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.592101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.592066 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-home\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.592251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.592111 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-tmp-dir\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.592251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.592154 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdfb\" (UniqueName: \"kubernetes.io/projected/e9e1d983-6558-405b-ad3a-58f3121815e6-kube-api-access-zpdfb\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.592405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.592382 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-model-cache\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.592586 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.592548 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-home\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.592743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.592618 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-tmp-dir\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.594290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.594267 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-dshm\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.594831 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.594812 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e1d983-6558-405b-ad3a-58f3121815e6-tls-certs\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.648600 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.648575 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdfb\" (UniqueName: \"kubernetes.io/projected/e9e1d983-6558-405b-ad3a-58f3121815e6-kube-api-access-zpdfb\") pod \"distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:42.773087 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:42.773001 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:43.199169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:43.199134 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn"] Nov 02 20:33:43.200608 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:33:43.200578 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9e1d983_6558_405b_ad3a_58f3121815e6.slice/crio-4dfaf53708f8a8761b77e50c89dad9f11e5a497d68122674d9b014409c8cc1d1 WatchSource:0}: Error finding container 4dfaf53708f8a8761b77e50c89dad9f11e5a497d68122674d9b014409c8cc1d1: Status 404 returned error can't find the container with id 4dfaf53708f8a8761b77e50c89dad9f11e5a497d68122674d9b014409c8cc1d1 Nov 02 20:33:44.206098 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:44.206062 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" event={"ID":"e9e1d983-6558-405b-ad3a-58f3121815e6","Type":"ContainerStarted","Data":"0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9"} Nov 02 20:33:44.206558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:44.206108 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" event={"ID":"e9e1d983-6558-405b-ad3a-58f3121815e6","Type":"ContainerStarted","Data":"4dfaf53708f8a8761b77e50c89dad9f11e5a497d68122674d9b014409c8cc1d1"} Nov 02 20:33:44.316909 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:44.316862 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podStartSLOduration=2.316849765 podStartE2EDuration="2.316849765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:33:44.314538605 +0000 UTC m=+1096.679149811" watchObservedRunningTime="2026-11-02 20:33:44.316849765 +0000 UTC m=+1096.681460969" Nov 02 20:33:44.773773 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:44.773745 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:52.773399 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:52.773367 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:52.791072 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:52.791044 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:33:53.276355 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:33:53.276326 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:34:02.660070 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:34:02.660027 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:34:32.658313 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:34:32.658271 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:34:37.426881 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.426837 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct"] Nov 02 20:34:37.431572 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.431550 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.528412 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.528384 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a5cad4\"/\"distinct-2-w6-a5cad4-kserve-self-signed-certs\"" Nov 02 20:34:37.531520 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.531496 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct"] Nov 02 20:34:37.555272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.555239 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-tmp-dir\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.555414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.555317 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpj4s\" (UniqueName: \"kubernetes.io/projected/ada702ce-5647-4d89-b72e-4af927326568-kube-api-access-mpj4s\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.555488 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.555437 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-home\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.555488 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.555475 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada702ce-5647-4d89-b72e-4af927326568-tls-certs\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.555614 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.555565 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-model-cache\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.555614 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.555596 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-dshm\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.656958 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.656859 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-model-cache\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.656958 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.656911 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-dshm\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.657206 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.656962 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-tmp-dir\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.657206 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.656998 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpj4s\" (UniqueName: \"kubernetes.io/projected/ada702ce-5647-4d89-b72e-4af927326568-kube-api-access-mpj4s\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.657206 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.657044 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-home\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.657206 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.657083 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada702ce-5647-4d89-b72e-4af927326568-tls-certs\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.657442 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.657405 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-home\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.657506 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.657461 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-tmp-dir\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.657605 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.657584 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-model-cache\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.659507 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.659485 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-dshm\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.659507 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.659496 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada702ce-5647-4d89-b72e-4af927326568-tls-certs\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.715467 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.715440 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpj4s\" (UniqueName: \"kubernetes.io/projected/ada702ce-5647-4d89-b72e-4af927326568-kube-api-access-mpj4s\") pod \"distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:37.744185 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:37.744166 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:38.125057 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:38.125029 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct"] Nov 02 20:34:38.126775 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:34:38.126745 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podada702ce_5647_4d89_b72e_4af927326568.slice/crio-60eb67171cb7123cb4c72245b73422cf81a6b9b5c2ce1b04968ee400b86f128f WatchSource:0}: Error finding container 60eb67171cb7123cb4c72245b73422cf81a6b9b5c2ce1b04968ee400b86f128f: Status 404 returned error can't find the container with id 60eb67171cb7123cb4c72245b73422cf81a6b9b5c2ce1b04968ee400b86f128f Nov 02 20:34:38.578085 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:38.578046 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" event={"ID":"ada702ce-5647-4d89-b72e-4af927326568","Type":"ContainerStarted","Data":"f3c6d651f87c4f29d399ebb3a952051ba69f78bd5d426207d779c339012106cd"} Nov 02 20:34:38.578483 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:38.578102 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" event={"ID":"ada702ce-5647-4d89-b72e-4af927326568","Type":"ContainerStarted","Data":"60eb67171cb7123cb4c72245b73422cf81a6b9b5c2ce1b04968ee400b86f128f"} Nov 02 20:34:38.722160 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:38.722117 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podStartSLOduration=1.722105042 podStartE2EDuration="1.722105042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:34:38.722093727 +0000 UTC m=+1151.086704931" watchObservedRunningTime="2026-11-02 20:34:38.722105042 +0000 UTC m=+1151.086716247" Nov 02 20:34:38.745170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:38.745144 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:47.744954 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:47.744919 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:47.757831 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:47.757803 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:34:48.663762 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:34:48.663732 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:35:02.661604 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:35:02.661560 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:35:09.212424 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.212344 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86"] Nov 02 20:35:09.216634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.216615 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.247961 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.247928 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-dshm\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.248092 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.248019 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btzfg\" (UniqueName: \"kubernetes.io/projected/0a598db0-2671-43ed-8d81-6100f706066d-kube-api-access-btzfg\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.248092 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.248076 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-model-cache\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.248184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.248114 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-home\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.248293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.248275 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a598db0-2671-43ed-8d81-6100f706066d-tls-certs\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.248366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.248313 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-tmp-dir\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.311566 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.311533 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-c49693\"/\"distinct-2-w5-c49693-kserve-self-signed-certs\"" Nov 02 20:35:09.315163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.315135 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86"] Nov 02 20:35:09.349367 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.349333 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-home\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.349504 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.349482 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a598db0-2671-43ed-8d81-6100f706066d-tls-certs\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.349577 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.349519 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-tmp-dir\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.349661 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.349645 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-dshm\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.349768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.349735 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btzfg\" (UniqueName: \"kubernetes.io/projected/0a598db0-2671-43ed-8d81-6100f706066d-kube-api-access-btzfg\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.349829 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.349772 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-model-cache\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.351007 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.350867 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-home\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.351277 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.351256 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-model-cache\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.351523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.351507 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-tmp-dir\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.354205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.354166 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a598db0-2671-43ed-8d81-6100f706066d-tls-certs\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.354395 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.354371 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-dshm\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.407231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.407197 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btzfg\" (UniqueName: \"kubernetes.io/projected/0a598db0-2671-43ed-8d81-6100f706066d-kube-api-access-btzfg\") pod \"distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.529987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.529914 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:09.939208 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:09.939132 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86"] Nov 02 20:35:09.940872 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:35:09.940847 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a598db0_2671_43ed_8d81_6100f706066d.slice/crio-4b6bee3d238963a98f25cf65f8bc4266da55a2f90b718228912370e424b45a99 WatchSource:0}: Error finding container 4b6bee3d238963a98f25cf65f8bc4266da55a2f90b718228912370e424b45a99: Status 404 returned error can't find the container with id 4b6bee3d238963a98f25cf65f8bc4266da55a2f90b718228912370e424b45a99 Nov 02 20:35:10.823515 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:10.823482 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" event={"ID":"0a598db0-2671-43ed-8d81-6100f706066d","Type":"ContainerStarted","Data":"c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a"} Nov 02 20:35:10.823515 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:10.823521 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" event={"ID":"0a598db0-2671-43ed-8d81-6100f706066d","Type":"ContainerStarted","Data":"4b6bee3d238963a98f25cf65f8bc4266da55a2f90b718228912370e424b45a99"} Nov 02 20:35:10.934315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:10.934266 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podStartSLOduration=1.934250509 podStartE2EDuration="1.934250509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:10.932010667 +0000 UTC m=+1183.296621871" watchObservedRunningTime="2026-11-02 20:35:10.934250509 +0000 UTC m=+1183.298861714" Nov 02 20:35:11.530343 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:11.530312 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:19.530840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:19.530807 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:19.543747 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:19.543724 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:19.891122 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:19.891093 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:35:20.855297 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.855265 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq"] Nov 02 20:35:20.859262 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.859239 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.959329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.959294 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq"] Nov 02 20:35:20.990463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990421 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.990463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990474 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.990739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990628 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.990739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990692 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.990873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990789 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.990873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990826 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72pl\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-kube-api-access-c72pl\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.991253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990928 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.991253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990965 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:20.991253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:20.990994 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092388 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092360 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092388 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092390 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092574 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092439 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092574 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092460 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092708 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092602 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092708 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092666 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72pl\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-kube-api-access-c72pl\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092744 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092769 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092803 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092808 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.092993 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.092847 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.093046 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.093020 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.093167 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.093150 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.093270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.093254 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.094815 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.094794 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.094914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.094903 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.154325 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.154251 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72pl\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-kube-api-access-c72pl\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.154325 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.154315 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.172930 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.172910 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:21.182759 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.182736 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth"] Nov 02 20:35:21.187257 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.187236 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.261708 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.257692 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth"] Nov 02 20:35:21.296140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296107 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.296281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296144 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.296281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296168 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.296366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296286 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7lt\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-kube-api-access-qb7lt\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.296366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296338 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.296431 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296389 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636700c4-2720-4f52-8043-d15cead9abf4-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.296539 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296515 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.296587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296571 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636700c4-2720-4f52-8043-d15cead9abf4-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.296739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.296722 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.397848 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.397818 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.397848 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.397852 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398070 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.397877 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398070 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.397915 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7lt\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-kube-api-access-qb7lt\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398070 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.397942 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398070 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.397985 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636700c4-2720-4f52-8043-d15cead9abf4-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398070 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.398044 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.398087 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636700c4-2720-4f52-8043-d15cead9abf4-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.398188 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.398242 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398540 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.398307 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398540 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.398515 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.398891 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.398870 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.399033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.399013 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636700c4-2720-4f52-8043-d15cead9abf4-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.400687 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.400651 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636700c4-2720-4f52-8043-d15cead9abf4-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.400780 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.400737 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.454977 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.454927 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.455131 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.455113 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7lt\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-kube-api-access-qb7lt\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.509352 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.509314 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:21.596712 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.596070 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq"] Nov 02 20:35:21.596972 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:35:21.596932 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fb4a0b_a907_4170_8367_fb9fe4479d0c.slice/crio-34ab8ed7a442b9ca8015e6d5f9894fdce13864182ae279f116c31db72e545b3f WatchSource:0}: Error finding container 34ab8ed7a442b9ca8015e6d5f9894fdce13864182ae279f116c31db72e545b3f: Status 404 returned error can't find the container with id 34ab8ed7a442b9ca8015e6d5f9894fdce13864182ae279f116c31db72e545b3f Nov 02 20:35:21.602004 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.601975 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:21.602067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.602035 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:21.602067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.602061 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:21.889569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.889533 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth"] Nov 02 20:35:21.891039 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:35:21.891014 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636700c4_2720_4f52_8043_d15cead9abf4.slice/crio-cf96df2ae22029bdddd3779248fc9d5f5fa4a07db021c8e6f805b4789b5352de WatchSource:0}: Error finding container cf96df2ae22029bdddd3779248fc9d5f5fa4a07db021c8e6f805b4789b5352de: Status 404 returned error can't find the container with id cf96df2ae22029bdddd3779248fc9d5f5fa4a07db021c8e6f805b4789b5352de Nov 02 20:35:21.897403 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.897364 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:21.897530 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.897448 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:21.897530 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.897489 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:35:21.900287 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.900265 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" event={"ID":"a3fb4a0b-a907-4170-8367-fb9fe4479d0c","Type":"ContainerStarted","Data":"c8396d34d13cb40a105e3fdfb66b76c7093c862bbb3b5e38a703c0e980c94086"} Nov 02 20:35:21.900407 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:21.900318 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" event={"ID":"a3fb4a0b-a907-4170-8367-fb9fe4479d0c","Type":"ContainerStarted","Data":"34ab8ed7a442b9ca8015e6d5f9894fdce13864182ae279f116c31db72e545b3f"} Nov 02 20:35:22.014646 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:22.014571 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podStartSLOduration=2.01455223 podStartE2EDuration="2.01455223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:22.01254255 +0000 UTC m=+1194.377153754" watchObservedRunningTime="2026-11-02 20:35:22.01455223 +0000 UTC m=+1194.379163437" Nov 02 20:35:22.173870 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:22.173779 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:22.175911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:22.175882 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:35:22.176061 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:22.175939 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:35:22.929850 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:22.929811 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" event={"ID":"636700c4-2720-4f52-8043-d15cead9abf4","Type":"ContainerStarted","Data":"70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2"} Nov 02 20:35:22.930249 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:22.929902 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" event={"ID":"636700c4-2720-4f52-8043-d15cead9abf4","Type":"ContainerStarted","Data":"cf96df2ae22029bdddd3779248fc9d5f5fa4a07db021c8e6f805b4789b5352de"} Nov 02 20:35:23.050335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:23.050285 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" podStartSLOduration=2.050268941 podStartE2EDuration="2.050268941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:23.048534897 +0000 UTC m=+1195.413146102" watchObservedRunningTime="2026-11-02 20:35:23.050268941 +0000 UTC m=+1195.414880193" Nov 02 20:35:23.181167 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:23.181063 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:35:23.182532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:23.182500 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:35:23.513114 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:23.513018 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:23.522942 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:23.522881 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:35:23.522942 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:23.522929 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:35:24.176696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:24.175064 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:35:24.176696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:24.175130 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:35:24.511773 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:24.511038 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:35:24.511773 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:24.511133 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:35:25.179801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:25.179764 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:35:25.180418 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:25.180377 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:35:25.512297 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:25.511083 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:35:25.512297 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:25.511140 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:35:26.178046 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:26.178005 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:35:26.178248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:26.178102 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:35:26.512750 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:26.512087 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:35:26.512750 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:26.512146 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:35:27.184705 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:27.184123 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:35:27.184705 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:27.184207 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:35:27.511095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:27.511004 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:35:27.511382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:27.511350 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:35:28.173754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:28.173705 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 20:35:28.174198 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:28.173772 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 20:35:28.510692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:28.510599 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 20:35:28.510833 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:28.510662 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 20:35:29.235808 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:29.235779 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:29.236541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:29.236514 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:29.237290 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:29.237272 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:35:29.514841 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:29.514770 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:29.515151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:29.515132 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:29.515862 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:29.515845 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:35:32.660377 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:35:32.660339 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:35:45.956840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:45.956808 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2"] Nov 02 20:35:45.961518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:45.961497 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.016373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.016346 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2"] Nov 02 20:35:46.055110 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.055082 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680154c9-0736-41e8-87f2-48e18a51c1f6-tls-certs\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.055246 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.055124 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-dshm\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.055246 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.055191 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-tmp-dir\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.055329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.055291 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvj5\" (UniqueName: \"kubernetes.io/projected/680154c9-0736-41e8-87f2-48e18a51c1f6-kube-api-access-nfvj5\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.055329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.055320 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-home\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.055440 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.055419 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-model-cache\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.058695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.058636 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"openshift-service-ca.crt\"" Nov 02 20:35:46.058810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.058717 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"default-dockercfg-jl9z8\"" Nov 02 20:35:46.058810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.058722 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"distinct-2-w3-61cdc1-kserve-self-signed-certs\"" Nov 02 20:35:46.058810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.058719 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-61cdc1\"/\"kube-root-ca.crt\"" Nov 02 20:35:46.158277 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.158241 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-model-cache\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.158455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.158391 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680154c9-0736-41e8-87f2-48e18a51c1f6-tls-certs\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.158527 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.158481 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-dshm\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.158580 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.158533 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-tmp-dir\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.158665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.158647 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvj5\" (UniqueName: \"kubernetes.io/projected/680154c9-0736-41e8-87f2-48e18a51c1f6-kube-api-access-nfvj5\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.158785 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.158709 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-home\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.159188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.159150 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-home\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.159431 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.159412 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-model-cache\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.160028 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.160006 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-tmp-dir\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.162130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.162110 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-dshm\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.163293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.163274 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680154c9-0736-41e8-87f2-48e18a51c1f6-tls-certs\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.217365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.217304 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvj5\" (UniqueName: \"kubernetes.io/projected/680154c9-0736-41e8-87f2-48e18a51c1f6-kube-api-access-nfvj5\") pod \"distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.275229 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.275206 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:46.661714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:46.658763 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2"] Nov 02 20:35:46.661714 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:35:46.659077 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod680154c9_0736_41e8_87f2_48e18a51c1f6.slice/crio-2ff99f8f29674261229eeb0810d0dd910804a701add6851d6e3b7a7d9151a8ec WatchSource:0}: Error finding container 2ff99f8f29674261229eeb0810d0dd910804a701add6851d6e3b7a7d9151a8ec: Status 404 returned error can't find the container with id 2ff99f8f29674261229eeb0810d0dd910804a701add6851d6e3b7a7d9151a8ec Nov 02 20:35:47.160664 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:47.160619 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" event={"ID":"680154c9-0736-41e8-87f2-48e18a51c1f6","Type":"ContainerStarted","Data":"18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116"} Nov 02 20:35:47.161110 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:47.160696 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" event={"ID":"680154c9-0736-41e8-87f2-48e18a51c1f6","Type":"ContainerStarted","Data":"2ff99f8f29674261229eeb0810d0dd910804a701add6851d6e3b7a7d9151a8ec"} Nov 02 20:35:47.274609 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:47.274551 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podStartSLOduration=2.274532461 podStartE2EDuration="2.274532461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:35:47.270006445 +0000 UTC m=+1219.634617651" watchObservedRunningTime="2026-11-02 20:35:47.274532461 +0000 UTC m=+1219.639143669" Nov 02 20:35:47.275810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:47.275781 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:56.276725 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:56.276688 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:56.288973 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:56.288948 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:35:57.275612 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:35:57.275584 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:36:02.664735 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:36:02.664692 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:36:05.855204 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.855167 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn"] Nov 02 20:36:05.860723 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.860702 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917263 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917237 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917383 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917289 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917383 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917313 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917470 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917400 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917470 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917434 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917470 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917455 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917617 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917563 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917706 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917628 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtx9v\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-kube-api-access-wtx9v\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.917706 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.917664 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:05.959756 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:05.959732 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn"] Nov 02 20:36:06.019286 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019260 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019303 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtx9v\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-kube-api-access-wtx9v\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019340 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019495 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019463 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019561 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019543 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019616 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019578 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019717 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019625 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019709 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019714 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.019796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.019774 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.020065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.020038 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.020173 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.020063 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.020173 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.020096 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.020402 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.020372 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.021907 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.021886 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.022036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.022020 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.084253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.084224 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.097400 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.097351 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtx9v\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-kube-api-access-wtx9v\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.175503 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.175278 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:06.493498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.493422 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx"] Nov 02 20:36:06.498704 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.498666 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.555953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.555927 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx"] Nov 02 20:36:06.596334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.596303 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn"] Nov 02 20:36:06.606266 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.606226 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:06.606377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.606313 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:06.606377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.606355 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:06.628505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628473 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.628642 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628521 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.628642 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628628 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.628788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628659 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.628788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628712 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.628788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628728 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szh54\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-kube-api-access-szh54\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.628788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628763 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.628788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628785 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.629016 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.628836 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730432 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730401 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730596 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730489 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730596 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730509 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730596 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730569 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730600 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730616 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730632 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szh54\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-kube-api-access-szh54\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730684 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.730770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730706 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.731058 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.730866 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.731141 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.731118 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-credential-socket\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.731911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.731573 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-data\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.731911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.731768 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-certs\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.732066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.732050 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istiod-ca-cert\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.733600 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.733569 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-envoy\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.733754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.733626 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-podinfo\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.789278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.789221 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-token\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.789379 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.789331 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szh54\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-kube-api-access-szh54\") pod \"e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:06.813167 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:06.813149 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:07.238857 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:07.238826 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx"] Nov 02 20:36:07.240450 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:36:07.240421 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc79e0e3_64d1_4bbd_9bf8_dfb9ffe81cc1.slice/crio-2cd3bd5623b32bbed3311ec2189a78a537f06c2db9662cd807f49c0687502bfc WatchSource:0}: Error finding container 2cd3bd5623b32bbed3311ec2189a78a537f06c2db9662cd807f49c0687502bfc: Status 404 returned error can't find the container with id 2cd3bd5623b32bbed3311ec2189a78a537f06c2db9662cd807f49c0687502bfc Nov 02 20:36:07.243505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:07.243475 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:07.243586 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:07.243537 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:07.243586 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:07.243563 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:07.331575 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:07.331545 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" event={"ID":"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1","Type":"ContainerStarted","Data":"2cd3bd5623b32bbed3311ec2189a78a537f06c2db9662cd807f49c0687502bfc"} Nov 02 20:36:07.333253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:07.333227 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" event={"ID":"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553","Type":"ContainerStarted","Data":"ddd4bcdc8f5cc7d0a3c94a65f16316f8124cb34cd2368924ba927b7e8e36ab1f"} Nov 02 20:36:07.333338 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:07.333278 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" event={"ID":"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553","Type":"ContainerStarted","Data":"2d111ed1716c067c920be4397710ba2a591b62bba4ac82cd16bfb29afc320903"} Nov 02 20:36:07.450951 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:07.450898 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" podStartSLOduration=2.450884193 podStartE2EDuration="2.450884193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:07.448161908 +0000 UTC m=+1239.812773118" watchObservedRunningTime="2026-11-02 20:36:07.450884193 +0000 UTC m=+1239.815495397" Nov 02 20:36:08.180316 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:08.180261 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:08.186813 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:08.186783 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:36:08.187028 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:08.187000 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:36:08.342136 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:08.342097 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" event={"ID":"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1","Type":"ContainerStarted","Data":"3b8a0ec1db1b1e898f2a5a4227cc951786915e7887ba5d2fc547f3b709e1a3ce"} Nov 02 20:36:08.461746 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:08.458432 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" podStartSLOduration=2.458411918 podStartE2EDuration="2.458411918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:08.454875762 +0000 UTC m=+1240.819486965" watchObservedRunningTime="2026-11-02 20:36:08.458411918 +0000 UTC m=+1240.823023126" Nov 02 20:36:08.817762 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:08.815834 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:08.820028 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:08.819988 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:36:08.820177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:08.820044 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:36:09.181750 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:09.177981 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:36:09.181750 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:09.178039 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:36:09.823717 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:09.814584 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:36:09.823717 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:09.814641 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:36:10.179778 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:10.179004 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:36:10.179778 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:10.179064 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:36:10.817075 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:10.817026 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:36:10.817339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:10.817296 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:36:11.178459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:11.177983 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:36:11.178459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:11.178038 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:36:11.830847 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:11.830767 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:36:11.831533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:11.831157 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:36:12.181120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:12.181021 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:36:12.181734 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:12.181699 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:36:12.815208 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:12.815166 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:36:12.815350 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:12.815254 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:36:13.177010 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:13.176920 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 20:36:13.177303 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:13.177276 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 20:36:13.814171 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:13.814139 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 02 20:36:13.814530 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:13.814200 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 02 20:36:14.182327 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:14.182246 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:14.182817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:14.182700 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:14.183715 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:14.183691 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:36:14.817988 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:14.817961 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:14.818414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:14.818396 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:14.819108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:14.819093 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:36:15.913032 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:15.912998 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n"] Nov 02 20:36:15.917249 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:15.917231 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.017260 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.017232 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n"] Nov 02 20:36:16.055606 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.055573 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf245911-b48a-4645-94e1-10a64054febd-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.055768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.055743 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.055848 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.055831 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.055967 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.055943 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.056057 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.055995 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.056057 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.056047 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.056190 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.056169 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-589q2\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-kube-api-access-589q2\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.056230 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.056220 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.056302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.056280 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf245911-b48a-4645-94e1-10a64054febd-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157274 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-589q2\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-kube-api-access-589q2\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157316 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157333 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf245911-b48a-4645-94e1-10a64054febd-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157382 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf245911-b48a-4645-94e1-10a64054febd-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157441 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157510 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157582 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157624 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157692 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157721 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.157948 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.157889 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.158049 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.158025 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.158177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.158097 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.158177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.158163 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf245911-b48a-4645-94e1-10a64054febd-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.160019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.159993 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.160263 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.160244 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf245911-b48a-4645-94e1-10a64054febd-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.214223 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.214175 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.214336 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.214319 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-589q2\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-kube-api-access-589q2\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.230160 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.230140 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:16.656420 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.656391 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n"] Nov 02 20:36:16.657552 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:36:16.657517 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf245911_b48a_4645_94e1_10a64054febd.slice/crio-07b29134ee7f417fc7ff41366664fb166d684fb254915d148b0f46a94d6dbd42 WatchSource:0}: Error finding container 07b29134ee7f417fc7ff41366664fb166d684fb254915d148b0f46a94d6dbd42: Status 404 returned error can't find the container with id 07b29134ee7f417fc7ff41366664fb166d684fb254915d148b0f46a94d6dbd42 Nov 02 20:36:16.661416 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.661381 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:16.661555 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.661441 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:16.661555 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:16.661471 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:17.475393 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:17.475357 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" event={"ID":"cf245911-b48a-4645-94e1-10a64054febd","Type":"ContainerStarted","Data":"ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9"} Nov 02 20:36:17.476034 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:17.475417 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" event={"ID":"cf245911-b48a-4645-94e1-10a64054febd","Type":"ContainerStarted","Data":"07b29134ee7f417fc7ff41366664fb166d684fb254915d148b0f46a94d6dbd42"} Nov 02 20:36:17.697295 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:17.697237 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podStartSLOduration=2.697224449 podStartE2EDuration="2.697224449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:17.645717138 +0000 UTC m=+1250.010328342" watchObservedRunningTime="2026-11-02 20:36:17.697224449 +0000 UTC m=+1250.061835715" Nov 02 20:36:18.231113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:18.231081 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:18.232478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:18.232453 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:36:18.232608 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:18.232497 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:36:19.230593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:19.230559 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:36:19.231027 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:19.230617 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:36:20.230844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:20.230811 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:36:20.231300 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:20.230881 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:36:21.322053 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:21.321952 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:36:21.344720 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:21.322012 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:36:22.234193 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:22.234158 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:36:22.234475 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:22.234444 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:36:23.054695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.051895 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 20:36:23.234106 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.234069 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:36:23.234383 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.234351 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:36:23.291344 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.291301 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.359751 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.355752 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.554692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.550428 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:23.669108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.667154 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:36:23.787696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.781195 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Nov 02 20:36:23.916836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:23.916760 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.055045 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.052770 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.055045 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.053504 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:36:24.090790 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.090749 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 20:36:24.138695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.137744 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 02 20:36:24.155694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.152783 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.255114 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.255077 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:36:24.255397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.255363 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:36:24.283695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.270770 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:36:24.301629 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.301261 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.327844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.327759 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:36:24.410697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.371838 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:36:24.422695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.415158 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:36:24.590949 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.561965 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.590949 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.582863 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:36:24.759399 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.673227 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.763743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.763712 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:36:24.927347 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.927080 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:24.931702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:24.930963 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:36:25.075731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.054311 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:25.097085 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.096757 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:25.097568 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.097549 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:36:25.165693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.154052 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:25.165693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.155806 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:36:25.259756 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.253870 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 20:36:25.259756 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.253955 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 20:36:25.282844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.282780 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:25.290890 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.290864 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:36:25.403696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.379607 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:25.496693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.493291 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:25.496693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.493545 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:36:25.496693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.493596 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:36:25.501770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.501749 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:36:25.521362 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.516732 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:36:25.605307 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.603423 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:36:25.606826 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.606805 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:36:25.636214 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.636190 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:36:25.638147 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.638118 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:36:25.645255 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:25.645151 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:36:26.521415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:26.521378 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:26.521956 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:26.521935 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:26.522629 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:26.522605 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:36:31.015830 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.015797 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2"] Nov 02 20:36:31.021523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.021496 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097110 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097081 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5490c04a-3123-4317-8906-a95d09bd1c13-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097284 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097119 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097355 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097302 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097409 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097359 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097430 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097517 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xxt\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-kube-api-access-j8xxt\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097563 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5490c04a-3123-4317-8906-a95d09bd1c13-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097614 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.097783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.097750 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.120720 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.120700 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2"] Nov 02 20:36:31.199087 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199056 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199267 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199109 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199267 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199204 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5490c04a-3123-4317-8906-a95d09bd1c13-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199267 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199225 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199319 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199370 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199398 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199435 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xxt\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-kube-api-access-j8xxt\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199481 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5490c04a-3123-4317-8906-a95d09bd1c13-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199555 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-data\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199663 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-credential-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199702 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-certs\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.199964 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.199943 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-socket\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.200574 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.200554 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5490c04a-3123-4317-8906-a95d09bd1c13-istiod-ca-cert\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.201909 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.201887 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5490c04a-3123-4317-8906-a95d09bd1c13-istio-podinfo\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.202073 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.202056 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-envoy\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.261640 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.261615 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-istio-token\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.261825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.261809 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xxt\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-kube-api-access-j8xxt\") pod \"e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.335896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.335871 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:31.763984 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.763956 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2"] Nov 02 20:36:31.764608 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:36:31.764574 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5490c04a_3123_4317_8906_a95d09bd1c13.slice/crio-59383a399d7359c1ac32eb9a228f3bb71839b8f6d7190112f34e876618b632d2 WatchSource:0}: Error finding container 59383a399d7359c1ac32eb9a228f3bb71839b8f6d7190112f34e876618b632d2: Status 404 returned error can't find the container with id 59383a399d7359c1ac32eb9a228f3bb71839b8f6d7190112f34e876618b632d2 Nov 02 20:36:31.767645 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.767604 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:31.767829 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.767730 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:31.767829 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.767774 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:31.949145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.949108 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" event={"ID":"5490c04a-3123-4317-8906-a95d09bd1c13","Type":"ContainerStarted","Data":"cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e"} Nov 02 20:36:31.949330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:31.949172 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" event={"ID":"5490c04a-3123-4317-8906-a95d09bd1c13","Type":"ContainerStarted","Data":"59383a399d7359c1ac32eb9a228f3bb71839b8f6d7190112f34e876618b632d2"} Nov 02 20:36:32.065575 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:32.065515 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podStartSLOduration=2.065497805 podStartE2EDuration="2.065497805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:32.061708963 +0000 UTC m=+1264.426320169" watchObservedRunningTime="2026-11-02 20:36:32.065497805 +0000 UTC m=+1264.430109010" Nov 02 20:36:32.337086 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:32.336993 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:32.338725 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:32.338699 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:36:32.338848 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:32.338745 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:36:32.694890 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:36:32.694788 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:36:33.336533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:33.336498 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:36:33.336924 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:33.336546 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:36:34.336378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:34.336350 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:36:34.336537 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:34.336397 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:36:35.336626 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:35.336591 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:36:35.337043 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:35.336643 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:36:36.336794 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.336759 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:36:36.337186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.336811 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:36:36.421852 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.421815 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2"] Nov 02 20:36:36.426433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.426411 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.509956 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.509925 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.510131 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.509973 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.510131 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.509997 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.510131 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.510096 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.510316 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.510234 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.510316 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.510264 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77786272-402f-4862-96f1-872cfcd090a9-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.510316 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.510302 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.510453 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.510328 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgzxn\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-kube-api-access-vgzxn\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.510453 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.510371 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77786272-402f-4862-96f1-872cfcd090a9-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.532749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.532720 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2"] Nov 02 20:36:36.611485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611399 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77786272-402f-4862-96f1-872cfcd090a9-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.611485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611467 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.611657 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611516 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.611657 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611547 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.611657 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611594 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.611817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611757 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.611817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611782 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77786272-402f-4862-96f1-872cfcd090a9-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.611912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611847 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.611912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.611876 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgzxn\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-kube-api-access-vgzxn\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.612502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.612475 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.612896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.612867 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77786272-402f-4862-96f1-872cfcd090a9-istiod-ca-cert\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.613106 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.613085 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-credential-socket\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.613218 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.613135 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-data\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.613387 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.613365 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-certs\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.621385 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.621360 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-envoy\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.622478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.622436 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77786272-402f-4862-96f1-872cfcd090a9-istio-podinfo\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.690706 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.686750 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgzxn\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-kube-api-access-vgzxn\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.694190 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.694161 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-istio-token\") pod \"e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.740451 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.740431 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:36.992770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.992696 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz"] Nov 02 20:36:36.998056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:36.998036 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.098701 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.098641 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz"] Nov 02 20:36:37.119353 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119314 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzp8c\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-kube-api-access-bzp8c\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.119535 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119520 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.119575 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119545 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.119611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119601 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.119786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119766 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.119827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119810 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.119874 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119863 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.119907 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119888 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.119986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.119909 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221392 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221363 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221464 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221403 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221464 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221429 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221534 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221500 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221534 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221514 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221604 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221574 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221641 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221601 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221641 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221627 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.221719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.221666 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzp8c\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-kube-api-access-bzp8c\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.222495 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.222456 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-data\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.222648 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.222563 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-certs\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.222648 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.222564 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istiod-ca-cert\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.222856 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.222833 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-credential-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.222952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.222931 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-socket\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.224024 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.223998 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2"] Nov 02 20:36:37.224183 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:36:37.224162 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77786272_402f_4862_96f1_872cfcd090a9.slice/crio-f5bd5a231c2e15f0ba46d5b8d13c39b8a3ae7d70d1415fe32e9613e08010900c WatchSource:0}: Error finding container f5bd5a231c2e15f0ba46d5b8d13c39b8a3ae7d70d1415fe32e9613e08010900c: Status 404 returned error can't find the container with id f5bd5a231c2e15f0ba46d5b8d13c39b8a3ae7d70d1415fe32e9613e08010900c Nov 02 20:36:37.225363 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.225339 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-envoy\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.225855 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.225838 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-podinfo\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.227220 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.227190 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:37.227299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.227250 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:37.227299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.227279 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:37.289763 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.289732 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzp8c\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-kube-api-access-bzp8c\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.289873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.289804 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-token\") pod \"e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.312643 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.312624 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:37.337216 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.337194 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:36:37.337618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.337237 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:36:37.800605 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.793255 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz"] Nov 02 20:36:37.803298 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.803259 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:37.803463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.803372 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:37.803463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:37.803414 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:38.021394 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.021304 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" event={"ID":"269eb374-e19c-4746-91ca-5f5a43ed1a1e","Type":"ContainerStarted","Data":"45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611"} Nov 02 20:36:38.021394 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.021357 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" event={"ID":"269eb374-e19c-4746-91ca-5f5a43ed1a1e","Type":"ContainerStarted","Data":"17645e1fe4c11d531c84c0877f5a3580b71bc0e6bcf1bb4bc033a5c6a13f9c23"} Nov 02 20:36:38.023768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.023742 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" event={"ID":"77786272-402f-4862-96f1-872cfcd090a9","Type":"ContainerStarted","Data":"8a51d14b809acc29ac976f946373e1a1ab795b411d451d8ca77549f17f763176"} Nov 02 20:36:38.023914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.023784 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" event={"ID":"77786272-402f-4862-96f1-872cfcd090a9","Type":"ContainerStarted","Data":"f5bd5a231c2e15f0ba46d5b8d13c39b8a3ae7d70d1415fe32e9613e08010900c"} Nov 02 20:36:38.142417 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.142354 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podStartSLOduration=2.142338175 podStartE2EDuration="2.142338175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:38.14095306 +0000 UTC m=+1270.505564299" watchObservedRunningTime="2026-11-02 20:36:38.142338175 +0000 UTC m=+1270.506949380" Nov 02 20:36:38.261740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.258497 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" podStartSLOduration=2.258465675 podStartE2EDuration="2.258465675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:38.25640971 +0000 UTC m=+1270.621020937" watchObservedRunningTime="2026-11-02 20:36:38.258465675 +0000 UTC m=+1270.623076881" Nov 02 20:36:38.313002 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.312966 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:38.315067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.314612 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:36:38.315067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.314705 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:36:38.336279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.336247 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:36:38.336439 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.336291 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:36:38.745433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.744593 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:38.745890 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.745858 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:36:38.746024 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:38.745898 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:36:39.320702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:39.320045 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:36:39.320702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:39.320105 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:36:39.428129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:39.424964 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:39.428129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:39.426422 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:39.428129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:39.427586 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:36:39.740925 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:39.740840 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:36:39.740925 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:39.740887 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:36:40.313181 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:40.313148 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:36:40.313546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:40.313202 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:36:40.741425 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:40.741344 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:36:40.741425 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:40.741392 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:36:41.313199 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:41.313161 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:36:41.313598 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:41.313216 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:36:41.740797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:41.740713 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:36:41.740797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:41.740763 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:36:42.313060 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:42.313013 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:36:42.313314 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:42.313286 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:36:42.741869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:42.741785 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:36:42.741869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:42.741848 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:36:43.313485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:43.313450 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:36:43.314120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:43.314088 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:36:43.741846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:43.741757 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 20:36:43.741846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:43.741830 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 20:36:44.313314 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:44.313274 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 02 20:36:44.313532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:44.313338 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 02 20:36:44.747088 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:44.747010 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:44.747320 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:44.747301 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:44.748194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:44.748177 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:36:45.318242 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:45.318210 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:45.318631 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:45.318611 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:45.319334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:45.319314 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:36:52.389351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.389316 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h"] Nov 02 20:36:52.393703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.393683 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.492329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.492288 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h"] Nov 02 20:36:52.520018 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.519988 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.520170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.520063 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.520170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.520125 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.520260 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.520223 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.520332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.520318 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.520396 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.520347 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.520484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.520466 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvpt\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-kube-api-access-wvvpt\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.520526 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.520493 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.520526 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.520522 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622157 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622128 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622307 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622191 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622307 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622234 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622307 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622275 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622480 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622357 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622480 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622456 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622662 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622641 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622791 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622635 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvpt\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-kube-api-access-wvvpt\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622791 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622731 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622791 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622738 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622810 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.622957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.622876 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.623061 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.623007 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.623104 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.623081 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.624786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.624765 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.625809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.625792 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.679801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.679744 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.679963 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.679944 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvpt\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-kube-api-access-wvvpt\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:52.708803 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:52.708778 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:53.184116 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:53.184087 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h"] Nov 02 20:36:53.184723 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:36:53.184665 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ffb888_80b2_42d3_9711_d8b09ffc5796.slice/crio-0bce8d37d3b9b458b32cb5c5a9d2315da495f29f2212eaa1f668e61ead5f992d WatchSource:0}: Error finding container 0bce8d37d3b9b458b32cb5c5a9d2315da495f29f2212eaa1f668e61ead5f992d: Status 404 returned error can't find the container with id 0bce8d37d3b9b458b32cb5c5a9d2315da495f29f2212eaa1f668e61ead5f992d Nov 02 20:36:53.187724 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:53.187648 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:53.187833 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:53.187747 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:53.187833 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:53.187778 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:36:53.217532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:53.217498 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" event={"ID":"d3ffb888-80b2-42d3-9711-d8b09ffc5796","Type":"ContainerStarted","Data":"0bce8d37d3b9b458b32cb5c5a9d2315da495f29f2212eaa1f668e61ead5f992d"} Nov 02 20:36:54.226443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:54.226406 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" event={"ID":"d3ffb888-80b2-42d3-9711-d8b09ffc5796","Type":"ContainerStarted","Data":"0b40e361510a7ce3a1137675341cc18b6149d86c837c8d3045c265cf552ce1a9"} Nov 02 20:36:54.348460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:54.348400 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podStartSLOduration=2.348383803 podStartE2EDuration="2.348383803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:36:54.346283719 +0000 UTC m=+1286.710894923" watchObservedRunningTime="2026-11-02 20:36:54.348383803 +0000 UTC m=+1286.712995009" Nov 02 20:36:54.709459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:54.709427 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:36:54.711113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:54.711084 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:36:54.711229 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:54.711137 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:36:55.867352 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:55.867317 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:36:55.867937 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:55.867904 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:36:56.498788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:56.498741 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:56.516691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:56.514737 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:56.616695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:56.613738 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:56.665696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:56.651741 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:56.665696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:56.657737 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:56.665696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:56.658003 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 20:36:56.731693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:56.717980 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:36:56.731693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:56.718040 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:36:57.276788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.276742 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.412812 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.412768 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.496862 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.496803 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.501828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.501742 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.502398 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.502380 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:36:57.503942 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.503915 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.521802 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.521767 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.523095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.522263 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:36:57.619504 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.619354 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.620495 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.620474 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:36:57.690099 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.690063 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.692703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.690659 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:36:57.732573 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.732540 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:36:57.732879 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.732843 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:36:57.741728 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.733936 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:57.765695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.742770 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:36:58.144698 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.143983 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.177694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:57.797583 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:36:58.178367 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.178335 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:36:58.425094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.424985 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:36:58.532695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.516994 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.607392 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.607347 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:36:58.622649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.622602 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.738066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.737956 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:58.741708 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.739238 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:36:58.747702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.743784 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:36:58.757933 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.757901 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 20:36:58.808700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.804554 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:36:58.808700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:58.804621 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:36:59.099695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.098872 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:59.155702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.137982 5420 trace.go:236] Trace[648523740]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-5984b9787d-hcg56" (02-Nov-2026 20:36:58.060) (total time: 1077ms): Nov 02 20:36:59.155702 ip-10-0-134-206 kubenswrapper[5420]: Trace[648523740]: [1.077045157s] [1.077045157s] END Nov 02 20:36:59.286023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.285986 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:36:59.286318 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.286292 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:59.393257 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.393158 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:36:59.440535 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.440497 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:59.527861 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.525766 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:59.635795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.635744 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 20:36:59.639597 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.638359 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Nov 02 20:36:59.694811 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.682756 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:59.699699 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:36:59.699106 5420 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.517s" Nov 02 20:36:59.699699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.699254 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:36:59.699699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.699291 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:36:59.699699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.699322 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:36:59.752096 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.752053 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:59.787818 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.776646 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:36:59.787818 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.776722 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:36:59.787818 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.778065 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 02 20:36:59.827695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.813790 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:36:59.835692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.835223 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:36:59.843573 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.843544 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:36:59.849223 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.849203 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:36:59.851439 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.851423 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:36:59.852436 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.852421 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:36:59.948740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.939822 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:36:59.948740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.941250 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:36:59.948740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.946136 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:36:59.953727 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:36:59.953706 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:37:00.713649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:00.713609 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:37:00.714169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:00.713689 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:37:01.710135 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:01.710097 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 20:37:01.710285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:01.710148 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 20:37:02.659865 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:37:02.657972 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:37:02.713184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:02.713161 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:37:02.713619 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:02.713607 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:37:02.714347 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:02.714329 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:37:21.418500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.418470 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf"] Nov 02 20:37:21.422870 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.422843 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.522581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.522548 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf"] Nov 02 20:37:21.529698 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.529651 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.529941 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.529914 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhv6\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-kube-api-access-pzhv6\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.530066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.529982 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.530066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.530031 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.530180 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.530081 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.530180 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.530143 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.530281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.530183 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.530281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.530222 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bddc36-c7cd-4413-8b70-d45e207a399a-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.530381 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.530303 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631021 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.630995 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631048 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhv6\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-kube-api-access-pzhv6\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631097 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631139 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631193 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631256 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631295 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631462 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631336 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bddc36-c7cd-4413-8b70-d45e207a399a-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631462 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631423 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631590 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631560 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-certs\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631590 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631578 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-data\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631664 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-credential-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.631825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631744 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-socket\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.632013 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.631994 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bddc36-c7cd-4413-8b70-d45e207a399a-istiod-ca-cert\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.633538 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.633518 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-envoy\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.633716 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.633698 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-podinfo\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.688321 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.688256 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-token\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.688788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.688768 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhv6\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-kube-api-access-pzhv6\") pod \"e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:21.743256 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:21.743234 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:22.181983 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:22.181948 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf"] Nov 02 20:37:22.186965 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:37:22.186924 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9bddc36_c7cd_4413_8b70_d45e207a399a.slice/crio-613b35c5fb66d49b03e8a541396d93c235032574f6d1fe2b5c52b42a8c9d1818 WatchSource:0}: Error finding container 613b35c5fb66d49b03e8a541396d93c235032574f6d1fe2b5c52b42a8c9d1818: Status 404 returned error can't find the container with id 613b35c5fb66d49b03e8a541396d93c235032574f6d1fe2b5c52b42a8c9d1818 Nov 02 20:37:22.189613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:22.189578 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:22.189776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:22.189661 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:22.189776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:22.189748 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:22.378316 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:22.378285 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" event={"ID":"a9bddc36-c7cd-4413-8b70-d45e207a399a","Type":"ContainerStarted","Data":"1469422394fa16f8cfffc41651dd99ffe158f8b9b749f955a0c1fd53c9d442cc"} Nov 02 20:37:22.378516 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:22.378334 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" event={"ID":"a9bddc36-c7cd-4413-8b70-d45e207a399a","Type":"ContainerStarted","Data":"613b35c5fb66d49b03e8a541396d93c235032574f6d1fe2b5c52b42a8c9d1818"} Nov 02 20:37:22.495827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:22.495731 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" podStartSLOduration=1.495716471 podStartE2EDuration="1.495716471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:22.492137285 +0000 UTC m=+1314.856748491" watchObservedRunningTime="2026-11-02 20:37:22.495716471 +0000 UTC m=+1314.860327675" Nov 02 20:37:22.743708 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:22.743653 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:23.745073 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:23.745027 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:37:23.745626 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:23.745099 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:37:24.755695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:24.754781 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:37:24.755695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:24.754862 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:37:25.744373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:25.744340 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:37:25.744540 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:25.744407 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:37:26.743844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:26.743811 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:37:26.744265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:26.743867 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:37:27.743755 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:27.743724 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:37:27.743922 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:27.743770 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:37:28.743775 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:28.743740 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 02 20:37:28.743938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:28.743816 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 02 20:37:30.047705 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:30.037964 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:30.047705 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:30.039514 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:30.069694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:30.068940 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:37:31.671700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:31.670775 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.813696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:31.813274 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.898698 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:31.874139 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.929751 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:31.902462 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.970714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:31.903431 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.970714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:31.949918 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:31.970964 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:31.949954 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:37:31.978712 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:31.949983 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.452696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.452206 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:37:32.452696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.452271 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.719750 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.694173 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.719750 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.695010 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:37:32.809553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.774869 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.865696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.830083 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:32.917492 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.910710 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:37:32.918334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.918310 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:37:32.918523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.918508 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:37:32.985984 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:37:32.985887 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:37:33.007406 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.988990 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.027878 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.989032 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.027878 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.989059 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.027878 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.989089 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.027878 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:32.992978 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:33.052721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.041827 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:37:33.052721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.041899 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:37:33.052721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.041935 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:37:33.052721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.041968 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:37:33.052721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.042004 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:37:33.720697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.712492 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 20:37:33.787697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.782888 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": context deadline exceeded" Nov 02 20:37:33.787697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.783351 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 20:37:33.905696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.796772 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:37:33.905696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:33.796829 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.158696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.069173 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.158696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.069503 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:37:34.158696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.069722 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:37:34.158696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.069945 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:37:34.158696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.070184 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.158696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.070382 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 02 20:37:34.158696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.125766 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.285695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.230150 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.733694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.729764 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.819985 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.819937 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:34.863456 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:37:34.862627 5420 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.095s" Nov 02 20:37:34.863766 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:34.863745 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:37:35.082830 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.082781 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 20:37:35.086692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.083413 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:37:35.090691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.087175 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:37:35.091127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.091101 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:37:35.116130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.116087 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:37:35.116653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.116625 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 02 20:37:35.171486 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.171447 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:35.834695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.834306 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Nov 02 20:37:35.926696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.926273 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 20:37:35.943697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:35.929148 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 02 20:37:36.118748 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.115326 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.118748 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.116603 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 20:37:36.118748 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.116920 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:37:36.151047 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.150995 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.151707 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.151651 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:37:36.152449 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.152179 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:37:36.732718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.710887 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.931865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.931819 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:36.934601 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:36.934565 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 20:37:37.121895 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.121829 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:37.131858 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.131814 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:37.151841 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.151801 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:37.154183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.152150 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:37.154183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.152825 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:37.177422 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.177383 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:37.331393 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.331354 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Nov 02 20:37:37.742833 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.738459 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:37:37.938774 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.938733 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 20:37:37.987141 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.987117 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:37:37.987462 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.987448 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:37:37.987809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.987797 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:37:37.987994 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.987978 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:37:37.988257 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.988246 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:37:37.988733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.988711 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:37:37.988865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.988784 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:37:37.988865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.988838 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:37:37.991966 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:37.991943 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:37:38.402897 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:38.400818 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6"] Nov 02 20:37:38.402897 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:38.401095 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" containerID="cri-o://db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297" gracePeriod=30 Nov 02 20:37:38.407014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:38.406967 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27"] Nov 02 20:37:38.407200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:38.407177 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" containerID="cri-o://977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281" gracePeriod=30 Nov 02 20:37:43.735786 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.735752 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:37:43.773716 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.773657 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:37:43.853731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.853665 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-socket\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.853920 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.853754 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-token\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-token\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.853920 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.853794 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istiod-ca-cert\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.853920 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.853829 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-certs\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.854089 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.853933 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-socket" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.854089 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.853955 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-envoy\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.854089 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854049 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-data\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.854253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854082 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-kube-api-access-c2tst\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-kube-api-access-c2tst\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.854253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854112 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-certs" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.854253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854129 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-podinfo\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.854253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854195 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istiod-ca-cert" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:43.854253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854211 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-credential-socket\") pod \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\" (UID: \"7017b523-af4a-4f4a-9f82-c9faeb58ff7c\") " Nov 02 20:37:43.854639 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854478 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-credential-socket" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.854794 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854765 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-data" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.855113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854893 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.855113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854909 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.855113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854918 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.855113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854926 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.855113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.854934 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.856196 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.856177 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-token" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:43.856375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.856351 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-envoy" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.856558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.856544 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-podinfo" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:37:43.856718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.856696 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-kube-api-access-c2tst" pod "7017b523-af4a-4f4a-9f82-c9faeb58ff7c" (UID: "7017b523-af4a-4f4a-9f82-c9faeb58ff7c"). InnerVolumeSpecName "kube-api-access-c2tst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:43.956330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956252 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/606c3566-3158-4235-b01f-fce9d9205c69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c3566-3158-4235-b01f-fce9d9205c69-istiod-ca-cert\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956294 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-socket\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956385 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-credential-socket\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956433 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-data\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956503 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-kube-api-access-69hv5\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-kube-api-access-69hv5\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956579 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/606c3566-3158-4235-b01f-fce9d9205c69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c3566-3158-4235-b01f-fce9d9205c69-istio-podinfo\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956585 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-socket" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.956709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956607 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/606c3566-3158-4235-b01f-fce9d9205c69-istiod-ca-cert" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:37:43.956709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956699 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-istio-token\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-istio-token\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956716 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-credential-socket" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.956918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956742 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-certs\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956793 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-envoy\") pod \"606c3566-3158-4235-b01f-fce9d9205c69\" (UID: \"606c3566-3158-4235-b01f-fce9d9205c69\") " Nov 02 20:37:43.956918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956742 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-data" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.957114 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.956922 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-certs" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.957622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957370 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.957622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957395 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.957622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957535 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2tst\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-kube-api-access-c2tst\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.957622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957562 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.957622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957578 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.957622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957593 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/606c3566-3158-4235-b01f-fce9d9205c69-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.957622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957606 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.957622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957620 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7017b523-af4a-4f4a-9f82-c9faeb58ff7c-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.957970 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.957636 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:43.958837 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.958809 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/606c3566-3158-4235-b01f-fce9d9205c69-istio-podinfo" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:37:43.958933 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.958890 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-envoy" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:37:43.959155 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.959134 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-istio-token" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:43.959224 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:43.959162 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-kube-api-access-69hv5" pod "606c3566-3158-4235-b01f-fce9d9205c69" (UID: "606c3566-3158-4235-b01f-fce9d9205c69"). InnerVolumeSpecName "kube-api-access-69hv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:37:44.059258 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.059231 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hv5\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-kube-api-access-69hv5\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:44.059258 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.059258 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/606c3566-3158-4235-b01f-fce9d9205c69-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:44.059426 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.059267 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/606c3566-3158-4235-b01f-fce9d9205c69-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:44.059426 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.059276 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/606c3566-3158-4235-b01f-fce9d9205c69-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:37:44.522756 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.522723 5420 generic.go:356] "Generic (PLEG): container finished" podID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerID="db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297" exitCode=0 Nov 02 20:37:44.522914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.522784 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" Nov 02 20:37:44.522914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.522822 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" event={"ID":"7017b523-af4a-4f4a-9f82-c9faeb58ff7c","Type":"ContainerDied","Data":"db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297"} Nov 02 20:37:44.522914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.522869 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6" event={"ID":"7017b523-af4a-4f4a-9f82-c9faeb58ff7c","Type":"ContainerDied","Data":"1b5ac186dd63f49d1d5bb405a9f1a780b4ac18fde3188cb204fb76b430ce76db"} Nov 02 20:37:44.522914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.522896 5420 scope.go:122] "RemoveContainer" containerID="db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297" Nov 02 20:37:44.524465 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.524443 5420 generic.go:356] "Generic (PLEG): container finished" podID="606c3566-3158-4235-b01f-fce9d9205c69" containerID="977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281" exitCode=0 Nov 02 20:37:44.524553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.524489 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" Nov 02 20:37:44.524553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.524520 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" event={"ID":"606c3566-3158-4235-b01f-fce9d9205c69","Type":"ContainerDied","Data":"977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281"} Nov 02 20:37:44.524618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.524562 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27" event={"ID":"606c3566-3158-4235-b01f-fce9d9205c69","Type":"ContainerDied","Data":"e457f8bab857148d18e4d65668194c9b46f5c1ba38f1797a26ca05e6d0d012fe"} Nov 02 20:37:44.534845 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.534634 5420 scope.go:122] "RemoveContainer" containerID="db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297" Nov 02 20:37:44.534981 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:37:44.534954 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297\": container with ID starting with db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297 not found: ID does not exist" containerID="db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297" Nov 02 20:37:44.535081 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.534988 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297"} err="failed to get container status \"db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297\": rpc error: code = NotFound desc = could not find container \"db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297\": container with ID starting with db0a85fbbf237dcf285443ace5b93499c274cde647ee28a33a754f8f4642f297 not found: ID does not exist" Nov 02 20:37:44.535081 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.535006 5420 scope.go:122] "RemoveContainer" containerID="977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281" Nov 02 20:37:44.544252 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.544236 5420 scope.go:122] "RemoveContainer" containerID="977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281" Nov 02 20:37:44.544494 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:37:44.544473 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281\": container with ID starting with 977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281 not found: ID does not exist" containerID="977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281" Nov 02 20:37:44.544552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.544500 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281"} err="failed to get container status \"977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281\": rpc error: code = NotFound desc = could not find container \"977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281\": container with ID starting with 977b68fa0b69762e27770eca1cf68d0bc1b6549e59d6720fb02f7b0859482281 not found: ID does not exist" Nov 02 20:37:44.729248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.729175 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6"] Nov 02 20:37:44.775395 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.775363 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-85fc6"] Nov 02 20:37:44.830331 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.830297 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27"] Nov 02 20:37:44.834598 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:44.834572 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-s9k27"] Nov 02 20:37:45.779625 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:45.779592 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="606c3566-3158-4235-b01f-fce9d9205c69" path="/var/lib/kubelet/pods/606c3566-3158-4235-b01f-fce9d9205c69/volumes" Nov 02 20:37:45.780294 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:45.780275 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" path="/var/lib/kubelet/pods/7017b523-af4a-4f4a-9f82-c9faeb58ff7c/volumes" Nov 02 20:37:49.367517 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.367476 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p"] Nov 02 20:37:49.368128 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.368113 5420 cpu_manager.go:414] "containerMap: removing container" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" Nov 02 20:37:49.368128 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.368127 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" Nov 02 20:37:49.368248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.368170 5420 cpu_manager.go:414] "containerMap: removing container" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" Nov 02 20:37:49.368248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.368175 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" Nov 02 20:37:49.368342 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.368269 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7017b523-af4a-4f4a-9f82-c9faeb58ff7c" containerName="istio-proxy" Nov 02 20:37:49.368342 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.368282 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="606c3566-3158-4235-b01f-fce9d9205c69" containerName="istio-proxy" Nov 02 20:37:49.371964 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.371943 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:49.435386 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.435338 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkhxz\" (UniqueName: \"kubernetes.io/projected/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-kube-api-access-vkhxz\") pod \"payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p\" (UID: \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:49.435560 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.435491 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p\" (UID: \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:49.467514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.467492 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6c66f658-dockercfg-8ppxs\"" Nov 02 20:37:49.467629 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.467563 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6c66f658\"" Nov 02 20:37:49.472986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.472966 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p"] Nov 02 20:37:49.571743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.569230 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p\" (UID: \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:49.571743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.569605 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkhxz\" (UniqueName: \"kubernetes.io/projected/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-kube-api-access-vkhxz\") pod \"payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p\" (UID: \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:49.571743 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.570556 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p\" (UID: \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:49.603539 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.603497 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln"] Nov 02 20:37:49.642373 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.642289 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:49.644791 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.644767 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln"] Nov 02 20:37:49.676944 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.676914 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkhxz\" (UniqueName: \"kubernetes.io/projected/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-kube-api-access-vkhxz\") pod \"payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p\" (UID: \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:49.700233 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.691375 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:49.772286 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.772259 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4wt\" (UniqueName: \"kubernetes.io/projected/7008fc5f-85c5-4690-99f5-8bce9804ec38-kube-api-access-qg4wt\") pod \"payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln\" (UID: \"7008fc5f-85c5-4690-99f5-8bce9804ec38\") " pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:49.772484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.772297 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7008fc5f-85c5-4690-99f5-8bce9804ec38-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln\" (UID: \"7008fc5f-85c5-4690-99f5-8bce9804ec38\") " pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:49.874382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.874329 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4wt\" (UniqueName: \"kubernetes.io/projected/7008fc5f-85c5-4690-99f5-8bce9804ec38-kube-api-access-qg4wt\") pod \"payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln\" (UID: \"7008fc5f-85c5-4690-99f5-8bce9804ec38\") " pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:49.874382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.874382 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7008fc5f-85c5-4690-99f5-8bce9804ec38-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln\" (UID: \"7008fc5f-85c5-4690-99f5-8bce9804ec38\") " pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:49.875360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.875332 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7008fc5f-85c5-4690-99f5-8bce9804ec38-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln\" (UID: \"7008fc5f-85c5-4690-99f5-8bce9804ec38\") " pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:49.931851 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:49.931788 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4wt\" (UniqueName: \"kubernetes.io/projected/7008fc5f-85c5-4690-99f5-8bce9804ec38-kube-api-access-qg4wt\") pod \"payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln\" (UID: \"7008fc5f-85c5-4690-99f5-8bce9804ec38\") " pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:50.013477 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.013451 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:50.087804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.083410 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p"] Nov 02 20:37:50.384120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.384094 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln"] Nov 02 20:37:50.385981 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:37:50.385954 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7008fc5f_85c5_4690_99f5_8bce9804ec38.slice/crio-0e011703a6aeb9ad2394e53ba859e7b93d0d1dc3b28ab3e0dcba77191b5d0f8d WatchSource:0}: Error finding container 0e011703a6aeb9ad2394e53ba859e7b93d0d1dc3b28ab3e0dcba77191b5d0f8d: Status 404 returned error can't find the container with id 0e011703a6aeb9ad2394e53ba859e7b93d0d1dc3b28ab3e0dcba77191b5d0f8d Nov 02 20:37:50.643459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.643428 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" event={"ID":"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0","Type":"ContainerStarted","Data":"aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1"} Nov 02 20:37:50.643637 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.643478 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" event={"ID":"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0","Type":"ContainerStarted","Data":"da62d381ecaa0d59bcc47a536260133ad7eca0c2201a3e3f1ef9512829f4cdcd"} Nov 02 20:37:50.643637 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.643519 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:37:50.645174 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.645151 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" event={"ID":"7008fc5f-85c5-4690-99f5-8bce9804ec38","Type":"ContainerStarted","Data":"02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400"} Nov 02 20:37:50.645280 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.645186 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" event={"ID":"7008fc5f-85c5-4690-99f5-8bce9804ec38","Type":"ContainerStarted","Data":"0e011703a6aeb9ad2394e53ba859e7b93d0d1dc3b28ab3e0dcba77191b5d0f8d"} Nov 02 20:37:50.645280 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.645249 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:37:50.752871 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.752799 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" podStartSLOduration=1.7527892889999999 podStartE2EDuration="1.752789289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:50.751303892 +0000 UTC m=+1343.115915098" watchObservedRunningTime="2026-11-02 20:37:50.752789289 +0000 UTC m=+1343.117400524" Nov 02 20:37:50.861497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:50.861453 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" podStartSLOduration=1.8614411039999998 podStartE2EDuration="1.861441104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:50.858823792 +0000 UTC m=+1343.223434997" watchObservedRunningTime="2026-11-02 20:37:50.861441104 +0000 UTC m=+1343.226052348" Nov 02 20:37:52.106116 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.106084 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m"] Nov 02 20:37:52.117966 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.117942 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:52.166923 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.166893 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m"] Nov 02 20:37:52.213729 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.213636 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-574357e2\"" Nov 02 20:37:52.214278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.214258 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-574357e2-dockercfg-2xdzh\"" Nov 02 20:37:52.242777 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.237589 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfk6s\" (UniqueName: \"kubernetes.io/projected/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-kube-api-access-bfk6s\") pod \"payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m\" (UID: \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:52.242777 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.237666 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m\" (UID: \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:52.313171 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.308721 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v"] Nov 02 20:37:52.341700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.334512 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:52.345689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.342789 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfk6s\" (UniqueName: \"kubernetes.io/projected/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-kube-api-access-bfk6s\") pod \"payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m\" (UID: \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:52.345689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.342897 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m\" (UID: \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:52.345689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.343895 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m\" (UID: \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:52.371739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.361377 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v"] Nov 02 20:37:52.404719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.402635 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfk6s\" (UniqueName: \"kubernetes.io/projected/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-kube-api-access-bfk6s\") pod \"payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m\" (UID: \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:52.434375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.434342 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:52.443980 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.443949 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a30dba-14fb-4d7c-b356-52ff6348c1e1-plugins-config-volume\") pod \"payload-processing-e2e-derive-574357e2-776f754456-2ht8v\" (UID: \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\") " pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:52.444153 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.444135 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5gpm\" (UniqueName: \"kubernetes.io/projected/85a30dba-14fb-4d7c-b356-52ff6348c1e1-kube-api-access-g5gpm\") pod \"payload-processing-e2e-derive-574357e2-776f754456-2ht8v\" (UID: \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\") " pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:52.545703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.545644 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5gpm\" (UniqueName: \"kubernetes.io/projected/85a30dba-14fb-4d7c-b356-52ff6348c1e1-kube-api-access-g5gpm\") pod \"payload-processing-e2e-derive-574357e2-776f754456-2ht8v\" (UID: \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\") " pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:52.545841 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.545824 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a30dba-14fb-4d7c-b356-52ff6348c1e1-plugins-config-volume\") pod \"payload-processing-e2e-derive-574357e2-776f754456-2ht8v\" (UID: \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\") " pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:52.546700 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.546655 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a30dba-14fb-4d7c-b356-52ff6348c1e1-plugins-config-volume\") pod \"payload-processing-e2e-derive-574357e2-776f754456-2ht8v\" (UID: \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\") " pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:52.606088 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.606047 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5gpm\" (UniqueName: \"kubernetes.io/projected/85a30dba-14fb-4d7c-b356-52ff6348c1e1-kube-api-access-g5gpm\") pod \"payload-processing-e2e-derive-574357e2-776f754456-2ht8v\" (UID: \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\") " pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:52.703251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.703168 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:52.820208 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:37:52.819924 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ab0db7_0515_419b_89dc_8ca93d1e91a9.slice/crio-7398c423c2131a485a23ff86d25eca846df6a4157287aab9005f7e54ad084f0b WatchSource:0}: Error finding container 7398c423c2131a485a23ff86d25eca846df6a4157287aab9005f7e54ad084f0b: Status 404 returned error can't find the container with id 7398c423c2131a485a23ff86d25eca846df6a4157287aab9005f7e54ad084f0b Nov 02 20:37:52.824569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.824504 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m"] Nov 02 20:37:52.847010 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.846978 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t"] Nov 02 20:37:52.855605 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.855578 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.865922 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.862662 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b"] Nov 02 20:37:52.872594 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.872570 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.930491 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.930301 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t"] Nov 02 20:37:52.953714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.953589 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.953714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.953664 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.953938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.953776 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.953938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.953806 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.953938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.953862 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.953938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.953898 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fea91a2-c7af-4431-aa75-cf955eec57c6-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.953938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.953929 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.954194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.953983 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.954194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954023 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2cmg\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-kube-api-access-d2cmg\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.954194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954107 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.954194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954141 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.954194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954171 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.954466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954213 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.954466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954300 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twncv\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-kube-api-access-twncv\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.954466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954328 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.954466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954387 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.954466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954427 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:52.954466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.954456 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:52.956330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:52.956301 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b"] Nov 02 20:37:53.055384 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055355 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.055514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055396 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.055514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055416 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.055514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055444 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.055514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055481 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.055514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055514 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.055844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055537 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.055844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055592 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.055844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055624 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fea91a2-c7af-4431-aa75-cf955eec57c6-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.055844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055655 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.055844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055737 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.055844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055779 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2cmg\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-kube-api-access-d2cmg\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055853 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055867 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055903 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055915 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055913 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.055965 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.056020 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.056108 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twncv\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-kube-api-access-twncv\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.056140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.056141 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.056626 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.056393 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.056626 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.056397 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-data\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.056626 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.056525 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.056804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.056654 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.056973 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.056914 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-credential-socket\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.057152 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.057061 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fea91a2-c7af-4431-aa75-cf955eec57c6-istiod-ca-cert\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.057152 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.057087 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-certs\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.058285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.058262 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.058894 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.058869 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.059212 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.059192 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-podinfo\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.059355 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.059339 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-envoy\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.087543 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.087514 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v"] Nov 02 20:37:53.087633 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:37:53.087513 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a30dba_14fb_4d7c_b356_52ff6348c1e1.slice/crio-53c5ef323bf4967a649c2ea0d870d054ada9f0fdcc101adff25eccad821b51fd WatchSource:0}: Error finding container 53c5ef323bf4967a649c2ea0d870d054ada9f0fdcc101adff25eccad821b51fd: Status 404 returned error can't find the container with id 53c5ef323bf4967a649c2ea0d870d054ada9f0fdcc101adff25eccad821b51fd Nov 02 20:37:53.114487 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.114463 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.114827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.114633 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-token\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.114827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.114642 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2cmg\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-kube-api-access-d2cmg\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.114827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.114703 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twncv\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-kube-api-access-twncv\") pod \"e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.161446 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.159556 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245"] Nov 02 20:37:53.164520 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.164499 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.181270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.181251 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:53.201734 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.201709 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:53.241243 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.241219 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245"] Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.259729 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.259782 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.259838 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.259902 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.259942 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj4g8\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-kube-api-access-hj4g8\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.260000 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063bf910-ea3f-4d1c-ac5b-d862a108a343-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.260148 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.260263 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.262733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.260333 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.362348 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362315 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.362491 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362372 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj4g8\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-kube-api-access-hj4g8\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.362491 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362420 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063bf910-ea3f-4d1c-ac5b-d862a108a343-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.362602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362538 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.362652 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362631 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.362759 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362735 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.362888 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362869 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.362956 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362906 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.363007 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.362955 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.363462 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.363438 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-data\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.368133 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.364515 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063bf910-ea3f-4d1c-ac5b-d862a108a343-istiod-ca-cert\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.368133 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.364804 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-certs\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.368133 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.365152 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.368133 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.367056 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-podinfo\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.370037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.370010 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-envoy\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.376030 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.375973 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-credential-socket\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.420850 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.420793 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj4g8\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-kube-api-access-hj4g8\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.421575 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.421551 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-token\") pod \"e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.481862 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.481807 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:53.588612 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:37:53.588576 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0d5bf8_5843_4ba4_9652_e29a1f05c965.slice/crio-4eed4d796448413aa933c6930174a8c903cf8f41535efbc129a73eedd1f68752 WatchSource:0}: Error finding container 4eed4d796448413aa933c6930174a8c903cf8f41535efbc129a73eedd1f68752: Status 404 returned error can't find the container with id 4eed4d796448413aa933c6930174a8c903cf8f41535efbc129a73eedd1f68752 Nov 02 20:37:53.593145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.593107 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.593257 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.593192 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.593257 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.593235 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.600599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.600564 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t"] Nov 02 20:37:53.673331 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.673274 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.673527 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.673442 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.673602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.673530 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.693183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.692628 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b"] Nov 02 20:37:53.702036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.702006 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" event={"ID":"5fea91a2-c7af-4431-aa75-cf955eec57c6","Type":"ContainerStarted","Data":"d91b056b58111507aa3a10d34f71d82ece97836e647f23505ed38be85a8a852e"} Nov 02 20:37:53.710294 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.710267 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" event={"ID":"d9ab0db7-0515-419b-89dc-8ca93d1e91a9","Type":"ContainerStarted","Data":"af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d"} Nov 02 20:37:53.710414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.710320 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" event={"ID":"d9ab0db7-0515-419b-89dc-8ca93d1e91a9","Type":"ContainerStarted","Data":"7398c423c2131a485a23ff86d25eca846df6a4157287aab9005f7e54ad084f0b"} Nov 02 20:37:53.710414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.710371 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:37:53.714588 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.714563 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" event={"ID":"85a30dba-14fb-4d7c-b356-52ff6348c1e1","Type":"ContainerStarted","Data":"bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509"} Nov 02 20:37:53.714661 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.714612 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" event={"ID":"85a30dba-14fb-4d7c-b356-52ff6348c1e1","Type":"ContainerStarted","Data":"53c5ef323bf4967a649c2ea0d870d054ada9f0fdcc101adff25eccad821b51fd"} Nov 02 20:37:53.715829 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.715205 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:37:53.721017 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.720993 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" event={"ID":"8d0d5bf8-5843-4ba4-9652-e29a1f05c965","Type":"ContainerStarted","Data":"4eed4d796448413aa933c6930174a8c903cf8f41535efbc129a73eedd1f68752"} Nov 02 20:37:53.822383 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.822314 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" podStartSLOduration=1.8222952700000001 podStartE2EDuration="1.82229527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:53.81812815 +0000 UTC m=+1346.182739355" watchObservedRunningTime="2026-11-02 20:37:53.82229527 +0000 UTC m=+1346.186906476" Nov 02 20:37:53.861206 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.861174 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245"] Nov 02 20:37:53.863833 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.863795 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.863936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.863873 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.863936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.863911 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:37:53.930537 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:53.930483 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" podStartSLOduration=1.9304645310000002 podStartE2EDuration="1.930464531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:53.925652269 +0000 UTC m=+1346.290263475" watchObservedRunningTime="2026-11-02 20:37:53.930464531 +0000 UTC m=+1346.295075737" Nov 02 20:37:54.841942 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:54.841901 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" event={"ID":"8d0d5bf8-5843-4ba4-9652-e29a1f05c965","Type":"ContainerStarted","Data":"555e4899e19d1415c2882f190398d485410dcfddbc6203072fec4fec657bed7b"} Nov 02 20:37:54.846658 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:54.846628 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" event={"ID":"063bf910-ea3f-4d1c-ac5b-d862a108a343","Type":"ContainerStarted","Data":"b846640c82272b739f7d719e075740482e2687624b4f2e198d355c1adf5babe0"} Nov 02 20:37:54.846912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:54.846895 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" event={"ID":"063bf910-ea3f-4d1c-ac5b-d862a108a343","Type":"ContainerStarted","Data":"6bc2fee262f5f3ad20f581a0092265f2403d3f411ff6f4b519b44afbfb5436ac"} Nov 02 20:37:54.849222 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:54.849200 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" event={"ID":"5fea91a2-c7af-4431-aa75-cf955eec57c6","Type":"ContainerStarted","Data":"a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874"} Nov 02 20:37:55.041071 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.041004 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" podStartSLOduration=3.040985583 podStartE2EDuration="3.040985583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:55.034442258 +0000 UTC m=+1347.399053463" watchObservedRunningTime="2026-11-02 20:37:55.040985583 +0000 UTC m=+1347.405596794" Nov 02 20:37:55.151229 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.151113 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" podStartSLOduration=3.151097521 podStartE2EDuration="3.151097521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:55.14788 +0000 UTC m=+1347.512491209" watchObservedRunningTime="2026-11-02 20:37:55.151097521 +0000 UTC m=+1347.515708728" Nov 02 20:37:55.186692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.186052 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:37:55.202698 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.200194 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 20:37:55.202698 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.200250 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 20:37:55.219696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.215504 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:37:55.219696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.215646 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:37:55.219696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.215707 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:37:55.295695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.295339 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" podStartSLOduration=2.295316735 podStartE2EDuration="2.295316735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:37:55.27795177 +0000 UTC m=+1347.642562975" watchObservedRunningTime="2026-11-02 20:37:55.295316735 +0000 UTC m=+1347.659927940" Nov 02 20:37:55.490748 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.489963 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:37:55.490748 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.490089 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:37:55.490748 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:55.490135 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:37:56.186692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:56.186619 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 20:37:56.186692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:56.186690 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 20:37:56.202138 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:56.202107 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:37:56.202389 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:56.202364 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:37:56.490158 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:56.490065 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:37:56.491735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:56.491700 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:37:57.202695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:57.189072 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 20:37:57.202695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:57.189145 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 20:37:57.204078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:57.204045 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:37:57.204276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:57.204253 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:37:57.482381 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:57.482295 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:37:57.482688 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:57.482636 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:37:58.186774 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:58.182067 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 20:37:58.186774 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:58.182138 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 20:37:58.218695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:58.208843 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:37:58.218695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:58.208899 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:37:58.497752 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:58.496699 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:37:58.497752 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:58.496759 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:37:59.257760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:59.252912 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:37:59.257760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:59.254480 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 20:37:59.257760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:59.254520 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 20:37:59.257760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:59.255330 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:37:59.257760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:59.255368 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:37:59.490846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:59.490812 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:37:59.491132 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:37:59.491101 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:38:00.182015 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:00.181970 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 02 20:38:00.182202 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:00.182031 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 02 20:38:00.203395 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:00.203080 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 02 20:38:00.203395 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:00.203136 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 02 20:38:00.483279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:00.483181 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 02 20:38:00.483279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:00.483253 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 02 20:38:01.473819 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.465046 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:38:01.473819 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.465710 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:38:01.474623 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.474572 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:38:01.625857 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.625824 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:38:01.626404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.626383 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:38:01.627208 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.627187 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:38:01.657026 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.657007 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:38:01.657163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.657141 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:38:01.714336 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.713661 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:38:01.715599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.714745 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:38:01.715599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:01.715476 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:38:02.662075 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:38:02.662024 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:38:04.852483 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:04.852452 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:38:04.852936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:04.852524 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:38:05.682075 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.682041 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj"] Nov 02 20:38:05.689355 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.689330 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.711989 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.711731 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.712923 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.712901 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.713079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.713060 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.713227 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.713210 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a3c81d-7baa-4a47-b018-931d8e75c077-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.713354 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.713336 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.713548 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.713526 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.713777 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.713756 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hwn\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-kube-api-access-h9hwn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.738246 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.738092 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 20:38:05.738246 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.738175 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 20:38:05.738392 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.738254 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 20:38:05.798694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.797949 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qhb9c\"" Nov 02 20:38:05.834724 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.834611 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.843689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.835280 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.843689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.835327 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.843689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.835391 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a3c81d-7baa-4a47-b018-931d8e75c077-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.843689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.835435 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.843689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.835497 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.843689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.835559 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hwn\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-kube-api-access-h9hwn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.843689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.837257 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:05.896730 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:05.885352 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj"] Nov 02 20:38:06.021795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:06.019242 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:06.021795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:06.021586 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a3c81d-7baa-4a47-b018-931d8e75c077-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:06.022099 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:06.021886 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:06.022161 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:06.022112 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:06.108695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:06.107416 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:06.145806 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:06.120498 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hwn\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-kube-api-access-h9hwn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zsbnj\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:06.327705 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:06.325451 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:07.077277 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:07.077233 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:07.508018 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:07.507933 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj"] Nov 02 20:38:07.619699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:07.615890 5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:38:08.109099 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.109045 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.110684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.110640 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 02 20:38:08.122696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.111581 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.148523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.115827 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 20:38:08.227708 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.193642 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:08.248690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.211386 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.248690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.229256 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:08.747660 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.747620 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" event={"ID":"91a3c81d-7baa-4a47-b018-931d8e75c077","Type":"ContainerStarted","Data":"c1ab2a1d966dc3521a6e3b209a812e44baf7345fce659fc03bb7f18f9709b8cf"} Nov 02 20:38:08.860713 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.860646 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Nov 02 20:38:08.861280 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:08.861248 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.040769 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.023755 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.110828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.110778 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.141467 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.140790 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:38:09.141845 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.115731 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:38:09.159693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.140188 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:38:09.159693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.152487 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:38:09.164775 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.164739 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.217926 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.217890 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:38:09.252694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.248691 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.257816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.257753 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.267854 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.267829 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:38:09.268085 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.268067 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:38:09.292124 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.269149 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.344933 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.344903 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:38:09.524820 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.524781 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:09.908803 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:09.870709 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 20:38:10.034866 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.034823 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.406463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.398362 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 02 20:38:10.422654 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.422611 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:10.423351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.423322 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 20:38:10.424129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.424102 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:38:10.424949 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.424923 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:10.427371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.427342 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:38:10.773040 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.772949 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:38:10.922693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.886229 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": context deadline exceeded" Nov 02 20:38:11.023880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:10.922039 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.235696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.014111 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:38:11.257941 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.257901 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.258180 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:38:11.167884 5420 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.203s" Nov 02 20:38:11.258430 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.258414 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:38:11.279114 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.258656 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:38:11.305622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.305593 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:38:11.316090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.066343 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.430267 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.430221 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.456385 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.456348 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.457995 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.457937 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Nov 02 20:38:11.460572 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.460548 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.476354 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.476299 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:11.476877 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.476725 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": context deadline exceeded" Nov 02 20:38:11.484487 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.484449 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:38:11.786713 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:11.775812 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:12.016796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.016754 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 20:38:12.447166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.447135 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:38:12.510938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.487627 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:12.549217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.549192 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:38:12.552640 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.552618 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:38:12.583005 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.582862 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:38:12.590714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.590690 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:38:12.594723 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.592535 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 20:38:12.630715 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.627878 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:12.630715 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.630140 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:38:12.635404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.634264 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:38:12.651974 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.649409 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:38:12.651974 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:12.651183 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:38:15.144284 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:15.144255 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:15.144546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:15.144318 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:15.671183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:15.671151 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" event={"ID":"91a3c81d-7baa-4a47-b018-931d8e75c077","Type":"ContainerStarted","Data":"3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734"} Nov 02 20:38:15.671360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:15.671232 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:15.835586 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:15.835525 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" podStartSLOduration=3.312808255 podStartE2EDuration="10.835507786s" totalImagesPullingTime="7.522699531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:15.786269469 +0000 UTC m=+1368.150880675" watchObservedRunningTime="2026-11-02 20:38:15.835507786 +0000 UTC m=+1368.200118992" Nov 02 20:38:16.681866 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:16.681833 5420 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zsbnj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:38:16.682265 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:16.681889 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" podUID="91a3c81d-7baa-4a47-b018-931d8e75c077" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:38:19.681838 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:19.681798 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:38:23.409361 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.409328 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp"] Nov 02 20:38:23.413913 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.413889 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592041 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592012 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592196 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592130 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592259 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592222 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fzcx\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-kube-api-access-6fzcx\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592331 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592313 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592392 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592347 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592392 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592385 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592501 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592454 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592501 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592482 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.592601 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.592514 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.615630 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.615603 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp"] Nov 02 20:38:23.694236 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694156 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694236 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694190 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694423 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694254 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694423 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694377 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694542 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694520 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694602 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694714 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694682 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fzcx\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-kube-api-access-6fzcx\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694724 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694749 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694781 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.694859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694795 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.695215 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.694899 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.695215 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.695088 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.695498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.695474 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.696961 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.696936 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.697313 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.697292 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.752128 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.752101 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:23.752300 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:23.752285 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fzcx\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-kube-api-access-6fzcx\") pod \"maas-default-gateway-openshift-default-85544d4cd7-zd2pp\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:24.027790 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:24.027719 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:24.454294 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:24.454261 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp"] Nov 02 20:38:24.454967 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:38:24.454942 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44f81ce_47d4_4b30_ab50_90135d0b45d2.slice/crio-71983dfede3b899a0216012aee87d74b0f318085cd496beb46b0c5bc3e0b3213 WatchSource:0}: Error finding container 71983dfede3b899a0216012aee87d74b0f318085cd496beb46b0c5bc3e0b3213: Status 404 returned error can't find the container with id 71983dfede3b899a0216012aee87d74b0f318085cd496beb46b0c5bc3e0b3213 Nov 02 20:38:24.457467 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:24.457435 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:24.457578 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:24.457507 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:24.457578 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:24.457551 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:24.749188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:24.749101 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" event={"ID":"e44f81ce-47d4-4b30-ab50-90135d0b45d2","Type":"ContainerStarted","Data":"56dcd9f299e63dc3eb565ab28c7f2214099de50023d801984df20c8e062604fc"} Nov 02 20:38:24.749188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:24.749161 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" event={"ID":"e44f81ce-47d4-4b30-ab50-90135d0b45d2","Type":"ContainerStarted","Data":"71983dfede3b899a0216012aee87d74b0f318085cd496beb46b0c5bc3e0b3213"} Nov 02 20:38:24.862139 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:24.862085 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podStartSLOduration=1.8620727910000001 podStartE2EDuration="1.862072791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:24.860791243 +0000 UTC m=+1377.225402447" watchObservedRunningTime="2026-11-02 20:38:24.862072791 +0000 UTC m=+1377.226683995" Nov 02 20:38:25.027920 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:25.027823 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:25.030114 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:25.030072 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-zd2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:38:25.030283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:25.030160 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:38:26.028191 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:26.028157 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-zd2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:38:26.028638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:26.028208 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:38:27.028773 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:27.028737 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-zd2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:38:27.029156 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:27.028798 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:38:28.028073 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:28.028043 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-zd2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:38:28.028169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:28.028104 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:38:29.028412 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:29.028374 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-zd2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:38:29.028936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:29.028436 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:38:30.028363 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:30.028326 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-zd2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:38:30.028542 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:30.028389 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:38:31.028664 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:31.028621 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-zd2pp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 02 20:38:31.029185 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:31.028723 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 02 20:38:32.664965 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:38:32.664927 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:38:32.739374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:32.739339 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:32.739970 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:32.739947 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:32.741665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:32.741638 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:38:37.276809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.276756 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.436709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.434801 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.440696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.437136 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.440696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.437259 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.471690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.470760 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.551860 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.551753 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.598971 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.598932 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.658859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.658805 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:37.787550 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:37.787508 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.066837 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.066795 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.522552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.522459 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 02 20:38:38.525318 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.525293 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:38:38.588601 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.560081 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 02 20:38:38.613695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.561431 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 02 20:38:38.616518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.614493 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:38:38.616518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.612769 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.616518 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.615759 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:38:38.616881 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.562157 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Nov 02 20:38:38.617310 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.617120 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:38:38.670695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.628028 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:38:38.737266 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.674219 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.737266 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.676190 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:38:38.759948 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.759908 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded" Nov 02 20:38:38.806783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.806743 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:38:38.807787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.807769 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:38:38.983908 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:38.983827 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:38:39.056078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:39.056049 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:38:39.131683 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:39.131595 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:38:39.190435 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:39.190408 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:38:39.348296 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:39.346958 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:38:39.349232 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:39.349207 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:38:39.350224 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:39.350209 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:38:53.394766 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.394735 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf"] Nov 02 20:38:53.399262 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.399242 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.498294 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.498263 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf"] Nov 02 20:38:53.567897 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.567867 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.568062 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.567909 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.568062 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.567931 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbp65\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-kube-api-access-cbp65\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.568143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.568088 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.568143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.568122 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.568219 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.568142 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.568219 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.568171 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.568286 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.568249 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.568323 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.568287 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a9bf42-6667-4ed6-b824-b4b4f209f989-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669290 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669335 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669356 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbp65\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-kube-api-access-cbp65\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669418 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669441 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669458 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669479 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669525 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669573 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a9bf42-6667-4ed6-b824-b4b4f209f989-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669961 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669738 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669961 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669829 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.669961 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.669850 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.670087 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.670026 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.670364 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.670341 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a9bf42-6667-4ed6-b824-b4b4f209f989-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.671857 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.671834 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.672103 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.672086 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.748986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.748961 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:53.749099 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:53.749045 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbp65\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-kube-api-access-cbp65\") pod \"maas-default-gateway-openshift-default-85544d4cd7-6jhlf\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:54.013736 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:54.013631 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:54.442526 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:54.442496 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf"] Nov 02 20:38:54.443627 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:38:54.443587 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a9bf42_6667_4ed6_b824_b4b4f209f989.slice/crio-4a9c93fa3b382c9edb2d643050065aae29ec060830c51776799764dece142763 WatchSource:0}: Error finding container 4a9c93fa3b382c9edb2d643050065aae29ec060830c51776799764dece142763: Status 404 returned error can't find the container with id 4a9c93fa3b382c9edb2d643050065aae29ec060830c51776799764dece142763 Nov 02 20:38:54.445996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:54.445968 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:54.446065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:54.446044 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:54.446101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:54.446075 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:38:55.301770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:55.301730 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" event={"ID":"01a9bf42-6667-4ed6-b824-b4b4f209f989","Type":"ContainerStarted","Data":"f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad"} Nov 02 20:38:55.301966 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:55.301796 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" event={"ID":"01a9bf42-6667-4ed6-b824-b4b4f209f989","Type":"ContainerStarted","Data":"4a9c93fa3b382c9edb2d643050065aae29ec060830c51776799764dece142763"} Nov 02 20:38:55.484694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:55.480889 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" podStartSLOduration=2.480869745 podStartE2EDuration="2.480869745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:38:55.48004084 +0000 UTC m=+1407.844652046" watchObservedRunningTime="2026-11-02 20:38:55.480869745 +0000 UTC m=+1407.845480977" Nov 02 20:38:56.020815 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:56.020777 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:38:56.022413 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:56.022377 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6jhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:38:56.022546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:56.022434 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:38:57.014616 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:57.014554 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6jhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:38:57.015026 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:57.014640 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:38:58.014898 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:58.014868 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6jhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:38:58.015372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:58.014913 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:38:59.014127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:59.014098 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6jhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:38:59.014310 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:38:59.014172 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:39:00.014094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:00.014060 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6jhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:39:00.014462 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:00.014123 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:39:01.014533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:01.014493 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-6jhlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 02 20:39:01.014937 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:01.014556 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 02 20:39:02.661446 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:39:02.661405 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:39:02.714198 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:02.713364 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:39:02.714646 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:02.714621 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:39:02.715498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:02.715478 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:39:32.659800 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:39:32.659753 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:39:46.491967 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:46.491889 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln"] Nov 02 20:39:46.609253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:46.609225 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p"] Nov 02 20:39:51.492391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.492354 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" podUID="7008fc5f-85c5-4690-99f5-8bce9804ec38" containerName="payload-processing" containerID="cri-o://02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400" gracePeriod=25 Nov 02 20:39:51.609617 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.609508 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" podUID="ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" containerName="payload-pre-processing" containerID="cri-o://aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1" gracePeriod=25 Nov 02 20:39:51.654316 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:39:51.654282 5420 logging.go:55] [core] [Channel #1529 SubChannel #1531]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.85:50052", ServerName: "10.133.0.85:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.85:50052: connect: connection refused" Nov 02 20:39:51.654316 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:39:51.654301 5420 logging.go:55] [core] [Channel #1530 SubChannel #1532]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.84:50052", ServerName: "10.133.0.84:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.84:50052: connect: connection refused" Nov 02 20:39:51.804074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.804048 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:39:51.878900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.878865 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7008fc5f-85c5-4690-99f5-8bce9804ec38-kube-api-access-qg4wt\" (UniqueName: \"kubernetes.io/projected/7008fc5f-85c5-4690-99f5-8bce9804ec38-kube-api-access-qg4wt\") pod \"7008fc5f-85c5-4690-99f5-8bce9804ec38\" (UID: \"7008fc5f-85c5-4690-99f5-8bce9804ec38\") " Nov 02 20:39:51.879053 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.878995 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7008fc5f-85c5-4690-99f5-8bce9804ec38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7008fc5f-85c5-4690-99f5-8bce9804ec38-plugins-config-volume\") pod \"7008fc5f-85c5-4690-99f5-8bce9804ec38\" (UID: \"7008fc5f-85c5-4690-99f5-8bce9804ec38\") " Nov 02 20:39:51.879686 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.879639 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7008fc5f-85c5-4690-99f5-8bce9804ec38-plugins-config-volume" pod "7008fc5f-85c5-4690-99f5-8bce9804ec38" (UID: "7008fc5f-85c5-4690-99f5-8bce9804ec38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:51.885258 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.885230 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7008fc5f-85c5-4690-99f5-8bce9804ec38-kube-api-access-qg4wt" pod "7008fc5f-85c5-4690-99f5-8bce9804ec38" (UID: "7008fc5f-85c5-4690-99f5-8bce9804ec38"). InnerVolumeSpecName "kube-api-access-qg4wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:51.904425 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.904404 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:39:51.951525 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.951495 5420 generic.go:356] "Generic (PLEG): container finished" podID="ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" containerID="aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1" exitCode=0 Nov 02 20:39:51.951801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.951530 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" event={"ID":"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0","Type":"ContainerDied","Data":"aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1"} Nov 02 20:39:51.951801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.951558 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" Nov 02 20:39:51.951801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.951588 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" event={"ID":"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0","Type":"ContainerDied","Data":"da62d381ecaa0d59bcc47a536260133ad7eca0c2201a3e3f1ef9512829f4cdcd"} Nov 02 20:39:51.951801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.951612 5420 scope.go:122] "RemoveContainer" containerID="aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1" Nov 02 20:39:51.953297 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.953277 5420 generic.go:356] "Generic (PLEG): container finished" podID="7008fc5f-85c5-4690-99f5-8bce9804ec38" containerID="02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400" exitCode=0 Nov 02 20:39:51.953380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.953305 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" event={"ID":"7008fc5f-85c5-4690-99f5-8bce9804ec38","Type":"ContainerDied","Data":"02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400"} Nov 02 20:39:51.953380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.953368 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" event={"ID":"7008fc5f-85c5-4690-99f5-8bce9804ec38","Type":"ContainerDied","Data":"0e011703a6aeb9ad2394e53ba859e7b93d0d1dc3b28ab3e0dcba77191b5d0f8d"} Nov 02 20:39:51.953462 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.953330 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" Nov 02 20:39:51.964327 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.964312 5420 scope.go:122] "RemoveContainer" containerID="aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1" Nov 02 20:39:51.964585 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:39:51.964564 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1\": container with ID starting with aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1 not found: ID does not exist" containerID="aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1" Nov 02 20:39:51.964653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.964592 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1"} err="failed to get container status \"aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1\": rpc error: code = NotFound desc = could not find container \"aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1\": container with ID starting with aac9983df90beedbba9dd40f89a9b2fabecb74282bde71868f42eeb6547300a1 not found: ID does not exist" Nov 02 20:39:51.964653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.964609 5420 scope.go:122] "RemoveContainer" containerID="02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400" Nov 02 20:39:51.976133 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.976116 5420 scope.go:122] "RemoveContainer" containerID="02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400" Nov 02 20:39:51.976399 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:39:51.976383 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400\": container with ID starting with 02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400 not found: ID does not exist" containerID="02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400" Nov 02 20:39:51.976469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.976404 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400"} err="failed to get container status \"02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400\": rpc error: code = NotFound desc = could not find container \"02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400\": container with ID starting with 02333d618c0cf1bd2353b947204f0dd1164f31cc32116b25c30b3c704e70a400 not found: ID does not exist" Nov 02 20:39:51.980372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.980348 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-plugins-config-volume\") pod \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\" (UID: \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\") " Nov 02 20:39:51.980664 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.980639 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-kube-api-access-vkhxz\" (UniqueName: \"kubernetes.io/projected/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-kube-api-access-vkhxz\") pod \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\" (UID: \"ae4548f6-c167-4f36-b0a3-8ce69f44ecf0\") " Nov 02 20:39:51.980780 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.980667 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-plugins-config-volume" pod "ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" (UID: "ae4548f6-c167-4f36-b0a3-8ce69f44ecf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:39:51.981207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.981184 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4wt\" (UniqueName: \"kubernetes.io/projected/7008fc5f-85c5-4690-99f5-8bce9804ec38-kube-api-access-qg4wt\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:51.981303 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.981210 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:51.981303 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.981225 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7008fc5f-85c5-4690-99f5-8bce9804ec38-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:51.982712 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:51.982687 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-kube-api-access-vkhxz" pod "ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" (UID: "ae4548f6-c167-4f36-b0a3-8ce69f44ecf0"). InnerVolumeSpecName "kube-api-access-vkhxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:39:52.083253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:52.083206 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkhxz\" (UniqueName: \"kubernetes.io/projected/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0-kube-api-access-vkhxz\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:39:52.220827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:52.220794 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln"] Nov 02 20:39:52.225299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:52.225275 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln"] Nov 02 20:39:52.370503 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:52.370425 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p"] Nov 02 20:39:52.374614 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:52.374591 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p"] Nov 02 20:39:52.654077 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:52.653984 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-migrate-6c66f658-798b9d975f-jkw9p" podUID="ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.84:50052\" within 1s: context deadline exceeded" Nov 02 20:39:52.654452 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:52.653993 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-6c66f658-646c8b7c64-5pwln" podUID="7008fc5f-85c5-4690-99f5-8bce9804ec38" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.85:50052\" within 1s: context deadline exceeded" Nov 02 20:39:53.778059 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:53.778030 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7008fc5f-85c5-4690-99f5-8bce9804ec38" path="/var/lib/kubelet/pods/7008fc5f-85c5-4690-99f5-8bce9804ec38/volumes" Nov 02 20:39:53.778510 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:53.778495 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" path="/var/lib/kubelet/pods/ae4548f6-c167-4f36-b0a3-8ce69f44ecf0/volumes" Nov 02 20:39:54.987788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:54.987751 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v"] Nov 02 20:39:55.105248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:55.105212 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m"] Nov 02 20:39:59.988612 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:39:59.988561 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" podUID="85a30dba-14fb-4d7c-b356-52ff6348c1e1" containerName="payload-processing" containerID="cri-o://bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509" gracePeriod=25 Nov 02 20:40:00.106393 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.106364 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" podUID="d9ab0db7-0515-419b-89dc-8ca93d1e91a9" containerName="payload-pre-processing" containerID="cri-o://af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d" gracePeriod=25 Nov 02 20:40:00.272375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.272351 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:40:00.394817 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.394792 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:40:00.405104 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.405072 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a30dba-14fb-4d7c-b356-52ff6348c1e1-kube-api-access-g5gpm\" (UniqueName: \"kubernetes.io/projected/85a30dba-14fb-4d7c-b356-52ff6348c1e1-kube-api-access-g5gpm\") pod \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\" (UID: \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\") " Nov 02 20:40:00.405237 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.405163 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a30dba-14fb-4d7c-b356-52ff6348c1e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a30dba-14fb-4d7c-b356-52ff6348c1e1-plugins-config-volume\") pod \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\" (UID: \"85a30dba-14fb-4d7c-b356-52ff6348c1e1\") " Nov 02 20:40:00.405780 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.405727 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a30dba-14fb-4d7c-b356-52ff6348c1e1-plugins-config-volume" pod "85a30dba-14fb-4d7c-b356-52ff6348c1e1" (UID: "85a30dba-14fb-4d7c-b356-52ff6348c1e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:00.406364 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.406339 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a30dba-14fb-4d7c-b356-52ff6348c1e1-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:00.407815 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.407743 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a30dba-14fb-4d7c-b356-52ff6348c1e1-kube-api-access-g5gpm" pod "85a30dba-14fb-4d7c-b356-52ff6348c1e1" (UID: "85a30dba-14fb-4d7c-b356-52ff6348c1e1"). InnerVolumeSpecName "kube-api-access-g5gpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:00.507921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.507864 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-plugins-config-volume\") pod \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\" (UID: \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\") " Nov 02 20:40:00.508020 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.507976 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-kube-api-access-bfk6s\" (UniqueName: \"kubernetes.io/projected/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-kube-api-access-bfk6s\") pod \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\" (UID: \"d9ab0db7-0515-419b-89dc-8ca93d1e91a9\") " Nov 02 20:40:00.508293 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.508251 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-plugins-config-volume" pod "d9ab0db7-0515-419b-89dc-8ca93d1e91a9" (UID: "d9ab0db7-0515-419b-89dc-8ca93d1e91a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:40:00.508710 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.508659 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5gpm\" (UniqueName: \"kubernetes.io/projected/85a30dba-14fb-4d7c-b356-52ff6348c1e1-kube-api-access-g5gpm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:00.508808 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.508713 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:00.510225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.510203 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-kube-api-access-bfk6s" pod "d9ab0db7-0515-419b-89dc-8ca93d1e91a9" (UID: "d9ab0db7-0515-419b-89dc-8ca93d1e91a9"). InnerVolumeSpecName "kube-api-access-bfk6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:40:00.610276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:00.610250 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfk6s\" (UniqueName: \"kubernetes.io/projected/d9ab0db7-0515-419b-89dc-8ca93d1e91a9-kube-api-access-bfk6s\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:40:01.034399 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.034374 5420 generic.go:356] "Generic (PLEG): container finished" podID="d9ab0db7-0515-419b-89dc-8ca93d1e91a9" containerID="af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d" exitCode=0 Nov 02 20:40:01.034847 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.034416 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" Nov 02 20:40:01.034847 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.034460 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" event={"ID":"d9ab0db7-0515-419b-89dc-8ca93d1e91a9","Type":"ContainerDied","Data":"af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d"} Nov 02 20:40:01.034847 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.034506 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m" event={"ID":"d9ab0db7-0515-419b-89dc-8ca93d1e91a9","Type":"ContainerDied","Data":"7398c423c2131a485a23ff86d25eca846df6a4157287aab9005f7e54ad084f0b"} Nov 02 20:40:01.034847 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.034528 5420 scope.go:122] "RemoveContainer" containerID="af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d" Nov 02 20:40:01.036172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.036141 5420 generic.go:356] "Generic (PLEG): container finished" podID="85a30dba-14fb-4d7c-b356-52ff6348c1e1" containerID="bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509" exitCode=0 Nov 02 20:40:01.036285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.036175 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" Nov 02 20:40:01.036285 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.036262 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" event={"ID":"85a30dba-14fb-4d7c-b356-52ff6348c1e1","Type":"ContainerDied","Data":"bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509"} Nov 02 20:40:01.036386 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.036299 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v" event={"ID":"85a30dba-14fb-4d7c-b356-52ff6348c1e1","Type":"ContainerDied","Data":"53c5ef323bf4967a649c2ea0d870d054ada9f0fdcc101adff25eccad821b51fd"} Nov 02 20:40:01.047366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.047351 5420 scope.go:122] "RemoveContainer" containerID="af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d" Nov 02 20:40:01.047694 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:40:01.047636 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d\": container with ID starting with af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d not found: ID does not exist" containerID="af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d" Nov 02 20:40:01.047792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.047709 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d"} err="failed to get container status \"af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d\": rpc error: code = NotFound desc = could not find container \"af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d\": container with ID starting with af9fbb4a73115e813c60ffe6a2e3c454ca64f1171c5fb16803df7cdca80b069d not found: ID does not exist" Nov 02 20:40:01.047792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.047735 5420 scope.go:122] "RemoveContainer" containerID="bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509" Nov 02 20:40:01.057117 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.057099 5420 scope.go:122] "RemoveContainer" containerID="bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509" Nov 02 20:40:01.057355 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:40:01.057336 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509\": container with ID starting with bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509 not found: ID does not exist" containerID="bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509" Nov 02 20:40:01.057424 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.057361 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509"} err="failed to get container status \"bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509\": rpc error: code = NotFound desc = could not find container \"bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509\": container with ID starting with bbd2e87512363e62999d527fe74de527dd9a5058e2e18d2b771a26e8de589509 not found: ID does not exist" Nov 02 20:40:01.189261 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.189235 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m"] Nov 02 20:40:01.234839 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.234809 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-574357e2-79899c95bc-fsl9m"] Nov 02 20:40:01.296663 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.296603 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v"] Nov 02 20:40:01.302328 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.302306 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-574357e2-776f754456-2ht8v"] Nov 02 20:40:01.776045 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.776018 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a30dba-14fb-4d7c-b356-52ff6348c1e1" path="/var/lib/kubelet/pods/85a30dba-14fb-4d7c-b356-52ff6348c1e1/volumes" Nov 02 20:40:01.776422 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:40:01.776408 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ab0db7-0515-419b-89dc-8ca93d1e91a9" path="/var/lib/kubelet/pods/d9ab0db7-0515-419b-89dc-8ca93d1e91a9/volumes" Nov 02 20:40:02.666531 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:40:02.666494 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:40:32.674391 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:40:32.674282 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:41:02.657536 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:41:02.657497 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:41:11.167405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.167329 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg"] Nov 02 20:41:11.168085 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168069 5420 cpu_manager.go:414] "containerMap: removing container" podUID="85a30dba-14fb-4d7c-b356-52ff6348c1e1" containerName="payload-processing" Nov 02 20:41:11.168130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168086 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a30dba-14fb-4d7c-b356-52ff6348c1e1" containerName="payload-processing" Nov 02 20:41:11.168130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168100 5420 cpu_manager.go:414] "containerMap: removing container" podUID="7008fc5f-85c5-4690-99f5-8bce9804ec38" containerName="payload-processing" Nov 02 20:41:11.168130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168105 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7008fc5f-85c5-4690-99f5-8bce9804ec38" containerName="payload-processing" Nov 02 20:41:11.168217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168145 5420 cpu_manager.go:414] "containerMap: removing container" podUID="d9ab0db7-0515-419b-89dc-8ca93d1e91a9" containerName="payload-pre-processing" Nov 02 20:41:11.168217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168150 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ab0db7-0515-419b-89dc-8ca93d1e91a9" containerName="payload-pre-processing" Nov 02 20:41:11.168217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168160 5420 cpu_manager.go:414] "containerMap: removing container" podUID="ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" containerName="payload-pre-processing" Nov 02 20:41:11.168217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168165 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" containerName="payload-pre-processing" Nov 02 20:41:11.168353 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168255 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ab0db7-0515-419b-89dc-8ca93d1e91a9" containerName="payload-pre-processing" Nov 02 20:41:11.168353 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168264 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4548f6-c167-4f36-b0a3-8ce69f44ecf0" containerName="payload-pre-processing" Nov 02 20:41:11.168353 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168272 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a30dba-14fb-4d7c-b356-52ff6348c1e1" containerName="payload-processing" Nov 02 20:41:11.168353 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.168278 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7008fc5f-85c5-4690-99f5-8bce9804ec38" containerName="payload-processing" Nov 02 20:41:11.171999 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.171982 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.268682 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.268645 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-23d588ad-serving-cert\"" Nov 02 20:41:11.272652 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.272629 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg"] Nov 02 20:41:11.311230 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.311201 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641a1c3-5451-41d1-8d94-d0c48ab8b117-openshift-service-ca\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.311376 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.311235 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.311452 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.311437 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-metrics-certs\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.311546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.311530 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjv6\" (UniqueName: \"kubernetes.io/projected/0641a1c3-5451-41d1-8d94-d0c48ab8b117-kube-api-access-rvjv6\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.412357 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.412324 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-metrics-certs\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.412521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.412417 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjv6\" (UniqueName: \"kubernetes.io/projected/0641a1c3-5451-41d1-8d94-d0c48ab8b117-kube-api-access-rvjv6\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.412521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.412508 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641a1c3-5451-41d1-8d94-d0c48ab8b117-openshift-service-ca\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.412615 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.412523 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.412722 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:41:11.412707 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-serving-cert: secret "maas-api-e2e-ait-23d588ad-serving-cert" not found Nov 02 20:41:11.412785 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:41:11.412774 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls podName:0641a1c3-5451-41d1-8d94-d0c48ab8b117 nodeName:}" failed. No retries permitted until 2026-11-02 20:41:11.912752965 +0000 UTC m=+1544.277364160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls") pod "maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" (UID: "0641a1c3-5451-41d1-8d94-d0c48ab8b117") : secret "maas-api-e2e-ait-23d588ad-serving-cert" not found Nov 02 20:41:11.413194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.413168 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641a1c3-5451-41d1-8d94-d0c48ab8b117-openshift-service-ca\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.414742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.414710 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-metrics-certs\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.471735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.471618 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjv6\" (UniqueName: \"kubernetes.io/projected/0641a1c3-5451-41d1-8d94-d0c48ab8b117-kube-api-access-rvjv6\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.918992 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.918953 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:11.921399 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:11.921370 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls\") pod \"maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:12.089886 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:12.089858 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:12.499715 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:12.499684 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg"] Nov 02 20:41:12.500706 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:41:12.500657 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0641a1c3_5451_41d1_8d94_d0c48ab8b117.slice/crio-ae90f5760d1481efbe6261d1596febb735a43a023480ce968861d0618ad3b878 WatchSource:0}: Error finding container ae90f5760d1481efbe6261d1596febb735a43a023480ce968861d0618ad3b878: Status 404 returned error can't find the container with id ae90f5760d1481efbe6261d1596febb735a43a023480ce968861d0618ad3b878 Nov 02 20:41:12.732933 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:12.732903 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" event={"ID":"0641a1c3-5451-41d1-8d94-d0c48ab8b117","Type":"ContainerStarted","Data":"ae90f5760d1481efbe6261d1596febb735a43a023480ce968861d0618ad3b878"} Nov 02 20:41:13.742109 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:13.742028 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" event={"ID":"0641a1c3-5451-41d1-8d94-d0c48ab8b117","Type":"ContainerStarted","Data":"b9830f2dbadb641e73b67666c449b9e64f664da42d7f3104f704fe3cb4989867"} Nov 02 20:41:13.742501 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:13.742173 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:13.853823 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:13.853773 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" podStartSLOduration=2.041511611 podStartE2EDuration="2.853756496s" totalImagesPullingTime="812.244885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:41:13.851330201 +0000 UTC m=+1546.215941406" watchObservedRunningTime="2026-11-02 20:41:13.853756496 +0000 UTC m=+1546.218367701" Nov 02 20:41:19.755722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:19.755694 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:41:20.535357 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.535314 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz"] Nov 02 20:41:20.574000 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.573969 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:20.632837 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.632810 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bb3d8f2f\"" Nov 02 20:41:20.633309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.633276 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bb3d8f2f-dockercfg-lhjh5\"" Nov 02 20:41:20.645008 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.644961 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz"] Nov 02 20:41:20.662856 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.662797 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f68223-bd3d-496b-a498-be15cc8bdec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz\" (UID: \"52f68223-bd3d-496b-a498-be15cc8bdec7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:20.663029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.662864 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58dh\" (UniqueName: \"kubernetes.io/projected/52f68223-bd3d-496b-a498-be15cc8bdec7-kube-api-access-t58dh\") pod \"payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz\" (UID: \"52f68223-bd3d-496b-a498-be15cc8bdec7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:20.765175 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.765134 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f68223-bd3d-496b-a498-be15cc8bdec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz\" (UID: \"52f68223-bd3d-496b-a498-be15cc8bdec7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:20.765664 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.765210 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58dh\" (UniqueName: \"kubernetes.io/projected/52f68223-bd3d-496b-a498-be15cc8bdec7-kube-api-access-t58dh\") pod \"payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz\" (UID: \"52f68223-bd3d-496b-a498-be15cc8bdec7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:20.766106 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.766079 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f68223-bd3d-496b-a498-be15cc8bdec7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz\" (UID: \"52f68223-bd3d-496b-a498-be15cc8bdec7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:20.827386 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.827350 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58dh\" (UniqueName: \"kubernetes.io/projected/52f68223-bd3d-496b-a498-be15cc8bdec7-kube-api-access-t58dh\") pod \"payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz\" (UID: \"52f68223-bd3d-496b-a498-be15cc8bdec7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:20.981693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:20.978328 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:21.409140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:21.409111 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz"] Nov 02 20:41:21.409921 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:41:21.409896 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f68223_bd3d_496b_a498_be15cc8bdec7.slice/crio-91d9d025a5600eae8b91e3e94b6fdcc1508bd67957719db8050629f31b3891f0 WatchSource:0}: Error finding container 91d9d025a5600eae8b91e3e94b6fdcc1508bd67957719db8050629f31b3891f0: Status 404 returned error can't find the container with id 91d9d025a5600eae8b91e3e94b6fdcc1508bd67957719db8050629f31b3891f0 Nov 02 20:41:21.827560 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:21.827527 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" event={"ID":"52f68223-bd3d-496b-a498-be15cc8bdec7","Type":"ContainerStarted","Data":"a67eae9c0b09be656ec4b27a7d8aa8f55142d2b6ca3c4bd376b48d8835055c70"} Nov 02 20:41:21.828003 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:21.827607 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" event={"ID":"52f68223-bd3d-496b-a498-be15cc8bdec7","Type":"ContainerStarted","Data":"91d9d025a5600eae8b91e3e94b6fdcc1508bd67957719db8050629f31b3891f0"} Nov 02 20:41:21.828003 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:21.827690 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:21.937953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:21.937903 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" podStartSLOduration=1.937890555 podStartE2EDuration="1.937890555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:41:21.934811079 +0000 UTC m=+1554.299422284" watchObservedRunningTime="2026-11-02 20:41:21.937890555 +0000 UTC m=+1554.302501777" Nov 02 20:41:25.533606 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.533563 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44"] Nov 02 20:41:25.538264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.538247 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.587234 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.587170 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-654ae4-openshift-default-dockercfg-wzv7s\"" Nov 02 20:41:25.639291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.639263 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44"] Nov 02 20:41:25.656525 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656499 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-socket\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.656666 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656551 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-podinfo\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.656666 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656628 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-certs\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.656768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656685 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-data\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.656768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656727 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-credential-socket\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.656768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656744 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-token\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.656768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656761 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8nz\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-kube-api-access-qp8nz\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.656916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656858 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istiod-ca-cert\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.656956 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.656945 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-envoy\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.758896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.758717 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-socket\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.775377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.771801 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-podinfo\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.775377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.771863 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-certs\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.775377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.771899 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-data\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.775377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.771938 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-credential-socket\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.775377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.771967 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-token\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.775377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.771992 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8nz\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-kube-api-access-qp8nz\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.775377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.772038 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istiod-ca-cert\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.775377 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.772153 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-envoy\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.781336 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.781304 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-credential-socket\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.782484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.782438 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-envoy\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.793783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.785553 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istiod-ca-cert\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.793783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.763121 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-socket\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.793783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.787584 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-certs\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.793783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.789548 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-podinfo\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.802366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.802342 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-data\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.837801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.837774 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-token\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.841410 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.841380 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8nz\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-kube-api-access-qp8nz\") pod \"e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:25.852453 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:25.852242 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:26.317074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:26.317043 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44"] Nov 02 20:41:26.317684 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:41:26.317642 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod035c4f0f_4f92_4ca6_8e3a_bc84ef17e73d.slice/crio-2aadaa341cc4fb859eac08ea0831c24e2a1b4155d5811b2c48744560f6487267 WatchSource:0}: Error finding container 2aadaa341cc4fb859eac08ea0831c24e2a1b4155d5811b2c48744560f6487267: Status 404 returned error can't find the container with id 2aadaa341cc4fb859eac08ea0831c24e2a1b4155d5811b2c48744560f6487267 Nov 02 20:41:26.320226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:26.320196 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:41:26.320301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:26.320255 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:41:26.320301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:26.320282 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:41:26.884025 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:26.883986 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" event={"ID":"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d","Type":"ContainerStarted","Data":"c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea"} Nov 02 20:41:26.884452 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:26.884044 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" event={"ID":"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d","Type":"ContainerStarted","Data":"2aadaa341cc4fb859eac08ea0831c24e2a1b4155d5811b2c48744560f6487267"} Nov 02 20:41:26.997947 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:26.997891 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" podStartSLOduration=1.997871967 podStartE2EDuration="1.997871967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:41:26.99598716 +0000 UTC m=+1559.360598367" watchObservedRunningTime="2026-11-02 20:41:26.997871967 +0000 UTC m=+1559.362483172" Nov 02 20:41:27.852916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:27.852888 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:27.858623 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:27.858599 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:27.891209 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:27.891183 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:27.891958 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:27.891941 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:41:32.660866 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:41:32.660830 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:41:32.838065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:32.838040 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:41:53.503178 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:53.503144 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz"] Nov 02 20:41:53.503644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:53.503320 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" containerID="cri-o://45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611" gracePeriod=30 Nov 02 20:41:58.788167 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.788143 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:41:58.915225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915192 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istiod-ca-cert\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.915366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915260 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-certs\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.915366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915292 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-token\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-token\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.915366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915321 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-kube-api-access-bzp8c\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-kube-api-access-bzp8c\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.915366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915363 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-data\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.915622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915573 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-envoy\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.915622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915572 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-certs" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.915622 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915593 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istiod-ca-cert" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:41:58.915816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915705 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-podinfo\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.915816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915716 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-data" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.915816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915750 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-credential-socket\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.915969 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.915855 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-socket\") pod \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\" (UID: \"269eb374-e19c-4746-91ca-5f5a43ed1a1e\") " Nov 02 20:41:58.916050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.916019 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-credential-socket" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.916186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.916161 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-socket" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.917422 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.917381 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.917642 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.917625 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.918080 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.918056 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.918170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.918082 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.918170 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.918096 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:58.919798 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.919774 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-token" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:58.919885 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.919812 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-kube-api-access-bzp8c" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "kube-api-access-bzp8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:41:58.920165 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.920144 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-envoy" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:41:58.924121 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:58.923030 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-podinfo" pod "269eb374-e19c-4746-91ca-5f5a43ed1a1e" (UID: "269eb374-e19c-4746-91ca-5f5a43ed1a1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:41:59.019382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.019308 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:59.019382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.019334 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:59.019382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.019343 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:59.019382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.019352 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzp8c\" (UniqueName: \"kubernetes.io/projected/269eb374-e19c-4746-91ca-5f5a43ed1a1e-kube-api-access-bzp8c\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:41:59.202650 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.202623 5420 generic.go:356] "Generic (PLEG): container finished" podID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerID="45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611" exitCode=0 Nov 02 20:41:59.202863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.202719 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" Nov 02 20:41:59.202863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.202717 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" event={"ID":"269eb374-e19c-4746-91ca-5f5a43ed1a1e","Type":"ContainerDied","Data":"45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611"} Nov 02 20:41:59.202863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.202847 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz" event={"ID":"269eb374-e19c-4746-91ca-5f5a43ed1a1e","Type":"ContainerDied","Data":"17645e1fe4c11d531c84c0877f5a3580b71bc0e6bcf1bb4bc033a5c6a13f9c23"} Nov 02 20:41:59.202863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.202862 5420 scope.go:122] "RemoveContainer" containerID="45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611" Nov 02 20:41:59.216543 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.216523 5420 scope.go:122] "RemoveContainer" containerID="45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611" Nov 02 20:41:59.216841 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:41:59.216818 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611\": container with ID starting with 45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611 not found: ID does not exist" containerID="45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611" Nov 02 20:41:59.216939 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.216852 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611"} err="failed to get container status \"45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611\": rpc error: code = NotFound desc = could not find container \"45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611\": container with ID starting with 45086d71e4a1bcd371ffd3f38a96a9843dc6e9f11ef76d9508216b2272a72611 not found: ID does not exist" Nov 02 20:41:59.360160 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.360126 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz"] Nov 02 20:41:59.411118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.411082 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-zwjwz"] Nov 02 20:41:59.777152 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:41:59.777060 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" path="/var/lib/kubelet/pods/269eb374-e19c-4746-91ca-5f5a43ed1a1e/volumes" Nov 02 20:42:01.140900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.140865 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2"] Nov 02 20:42:01.142036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.142011 5420 cpu_manager.go:414] "containerMap: removing container" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" Nov 02 20:42:01.142161 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.142037 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" Nov 02 20:42:01.142259 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.142246 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="269eb374-e19c-4746-91ca-5f5a43ed1a1e" containerName="istio-proxy" Nov 02 20:42:01.148124 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.148099 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.265928 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.265893 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2"] Nov 02 20:42:01.266351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.266331 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-0ea576-openshift-default-dockercfg-fv7vj\"" Nov 02 20:42:01.295991 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.295955 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-envoy\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.296485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.296463 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-certs\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.296757 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.296740 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-credential-socket\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.296957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.296940 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-token\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.297129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.297115 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-data\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.297302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.297287 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-kube-api-access-h8nkv\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.297458 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.297445 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-socket\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.297710 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.297692 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-podinfo\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.297872 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.297857 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istiod-ca-cert\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406178 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-podinfo\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406227 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istiod-ca-cert\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406303 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-envoy\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406351 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-certs\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406419 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-credential-socket\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406482 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-token\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406524 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-data\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406590 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-kube-api-access-h8nkv\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407095 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.406647 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-socket\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407685 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.407189 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-socket\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.407869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.407844 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-credential-socket\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.408517 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.408492 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istiod-ca-cert\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.415688 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.410165 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-data\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.415688 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.411166 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-certs\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.433688 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.432496 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-envoy\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.433688 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.433262 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-podinfo\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.468827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.468796 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-token\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.470728 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.470700 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-kube-api-access-h8nkv\") pod \"e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:01.768735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:01.766445 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:02.242447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:02.242417 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2"] Nov 02 20:42:02.243062 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:42:02.243040 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa56fb7_abaa_4169_9c0a_319feee7dc86.slice/crio-334c8784bc5b7f168c481d9892de98444861bb7da852cec97a530b0076013ab4 WatchSource:0}: Error finding container 334c8784bc5b7f168c481d9892de98444861bb7da852cec97a530b0076013ab4: Status 404 returned error can't find the container with id 334c8784bc5b7f168c481d9892de98444861bb7da852cec97a530b0076013ab4 Nov 02 20:42:02.249757 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:02.246484 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:42:02.249757 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:02.246644 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:42:02.249757 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:02.246746 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:42:02.667742 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:42:02.667706 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:42:03.254038 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:03.254002 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" event={"ID":"eaa56fb7-abaa-4169-9c0a-319feee7dc86","Type":"ContainerStarted","Data":"a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc"} Nov 02 20:42:03.254448 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:03.254088 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" event={"ID":"eaa56fb7-abaa-4169-9c0a-319feee7dc86","Type":"ContainerStarted","Data":"334c8784bc5b7f168c481d9892de98444861bb7da852cec97a530b0076013ab4"} Nov 02 20:42:03.413832 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:03.413785 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" podStartSLOduration=2.413771269 podStartE2EDuration="2.413771269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:03.412475694 +0000 UTC m=+1595.777086911" watchObservedRunningTime="2026-11-02 20:42:03.413771269 +0000 UTC m=+1595.778382474" Nov 02 20:42:03.778967 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:03.778939 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:03.798789 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:03.795045 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:04.263586 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:04.263551 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:04.264644 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:04.264626 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:42:18.091690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.085434 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z"] Nov 02 20:42:18.114623 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.114591 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:18.129690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.128738 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqnmg\" (UniqueName: \"kubernetes.io/projected/1e74e222-c059-4021-910f-78d9819015c2-kube-api-access-jqnmg\") pod \"payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z\" (UID: \"1e74e222-c059-4021-910f-78d9819015c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:18.129690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.128952 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e74e222-c059-4021-910f-78d9819015c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z\" (UID: \"1e74e222-c059-4021-910f-78d9819015c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:18.139747 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.139720 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z"] Nov 02 20:42:18.182209 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.181977 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-0ea576\"" Nov 02 20:42:18.182209 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.182173 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-0ea576-dockercfg-g7zfp\"" Nov 02 20:42:18.239694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.234374 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqnmg\" (UniqueName: \"kubernetes.io/projected/1e74e222-c059-4021-910f-78d9819015c2-kube-api-access-jqnmg\") pod \"payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z\" (UID: \"1e74e222-c059-4021-910f-78d9819015c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:18.239694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.234636 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e74e222-c059-4021-910f-78d9819015c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z\" (UID: \"1e74e222-c059-4021-910f-78d9819015c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:18.239694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.235744 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e74e222-c059-4021-910f-78d9819015c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z\" (UID: \"1e74e222-c059-4021-910f-78d9819015c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:18.301684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.301624 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqnmg\" (UniqueName: \"kubernetes.io/projected/1e74e222-c059-4021-910f-78d9819015c2-kube-api-access-jqnmg\") pod \"payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z\" (UID: \"1e74e222-c059-4021-910f-78d9819015c2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:18.445499 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.445397 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:18.829180 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:18.829154 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z"] Nov 02 20:42:18.830163 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:42:18.830137 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e74e222_c059_4021_910f_78d9819015c2.slice/crio-c09e788d18f5133a310d0d516e5e0832217456a1e6c2be0e73fca2c426e8ba92 WatchSource:0}: Error finding container c09e788d18f5133a310d0d516e5e0832217456a1e6c2be0e73fca2c426e8ba92: Status 404 returned error can't find the container with id c09e788d18f5133a310d0d516e5e0832217456a1e6c2be0e73fca2c426e8ba92 Nov 02 20:42:19.415459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:19.415427 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" event={"ID":"1e74e222-c059-4021-910f-78d9819015c2","Type":"ContainerStarted","Data":"e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717"} Nov 02 20:42:19.415898 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:19.415487 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" event={"ID":"1e74e222-c059-4021-910f-78d9819015c2","Type":"ContainerStarted","Data":"c09e788d18f5133a310d0d516e5e0832217456a1e6c2be0e73fca2c426e8ba92"} Nov 02 20:42:19.415898 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:19.415517 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:19.533654 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:19.533580 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" podStartSLOduration=1.53356169 podStartE2EDuration="1.53356169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:19.523052709 +0000 UTC m=+1611.887663913" watchObservedRunningTime="2026-11-02 20:42:19.53356169 +0000 UTC m=+1611.898172895" Nov 02 20:42:29.711086 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.711054 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c"] Nov 02 20:42:29.716842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.716823 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.797221 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.797082 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.797221 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.797172 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rscz\" (UniqueName: \"kubernetes.io/projected/0afab02f-589b-473a-91d8-35ee552bb40a-kube-api-access-2rscz\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.797463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.797411 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.797463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.797447 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afab02f-589b-473a-91d8-35ee552bb40a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.800663 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.800638 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c"] Nov 02 20:42:29.810078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.810053 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-654ae4-serving-cert\"" Nov 02 20:42:29.898606 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.898574 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.898811 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.898650 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rscz\" (UniqueName: \"kubernetes.io/projected/0afab02f-589b-473a-91d8-35ee552bb40a-kube-api-access-2rscz\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.898928 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.898897 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.899056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.898938 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afab02f-589b-473a-91d8-35ee552bb40a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.899056 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:42:29.899036 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-serving-cert: secret "maas-api-e2e-shared-a-w1-654ae4-serving-cert" not found Nov 02 20:42:29.899167 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:42:29.899126 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls podName:0afab02f-589b-473a-91d8-35ee552bb40a nodeName:}" failed. No retries permitted until 2026-11-02 20:42:30.399106177 +0000 UTC m=+1622.763717360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls") pod "maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" (UID: "0afab02f-589b-473a-91d8-35ee552bb40a") : secret "maas-api-e2e-shared-a-w1-654ae4-serving-cert" not found Nov 02 20:42:29.899612 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.899592 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afab02f-589b-473a-91d8-35ee552bb40a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.901033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.901012 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:29.960160 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:29.960098 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rscz\" (UniqueName: \"kubernetes.io/projected/0afab02f-589b-473a-91d8-35ee552bb40a-kube-api-access-2rscz\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:30.410043 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:30.410012 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:30.412336 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:30.412312 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:30.429308 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:30.429288 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:42:30.631552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:30.631519 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:31.031736 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:31.031706 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c"] Nov 02 20:42:31.033763 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:42:31.033731 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afab02f_589b_473a_91d8_35ee552bb40a.slice/crio-bfe9661be2a45b4ef64a3e67b94df1d78c891ceba431fe09032c8e04e65843bf WatchSource:0}: Error finding container bfe9661be2a45b4ef64a3e67b94df1d78c891ceba431fe09032c8e04e65843bf: Status 404 returned error can't find the container with id bfe9661be2a45b4ef64a3e67b94df1d78c891ceba431fe09032c8e04e65843bf Nov 02 20:42:31.514627 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:31.514592 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" event={"ID":"0afab02f-589b-473a-91d8-35ee552bb40a","Type":"ContainerStarted","Data":"bfe9661be2a45b4ef64a3e67b94df1d78c891ceba431fe09032c8e04e65843bf"} Nov 02 20:42:32.538313 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:32.538274 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" event={"ID":"0afab02f-589b-473a-91d8-35ee552bb40a","Type":"ContainerStarted","Data":"6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5"} Nov 02 20:42:32.539440 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:32.539413 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:32.655797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:32.655744 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" podStartSLOduration=2.837789475 podStartE2EDuration="3.655725671s" totalImagesPullingTime="817.936196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:42:32.652615419 +0000 UTC m=+1625.017226623" watchObservedRunningTime="2026-11-02 20:42:32.655725671 +0000 UTC m=+1625.020336876" Nov 02 20:42:32.676797 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:42:32.676757 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:42:39.575337 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:39.575262 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:42:53.394172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:53.394139 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9"] Nov 02 20:42:53.394809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:53.394342 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" containerID="cri-o://656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5" gracePeriod=30 Nov 02 20:42:53.440465 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:53.440432 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq"] Nov 02 20:42:53.440754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:53.440718 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" containerID="cri-o://c8396d34d13cb40a105e3fdfb66b76c7093c862bbb3b5e38a703c0e980c94086" gracePeriod=30 Nov 02 20:42:53.445070 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:53.445047 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2"] Nov 02 20:42:53.445255 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:53.445235 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" containerID="cri-o://8a51d14b809acc29ac976f946373e1a1ab795b411d451d8ca77549f17f763176" gracePeriod=30 Nov 02 20:42:57.088797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:57.088761 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:42:57.089330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:57.089301 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:42:58.809926 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.809878 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:42:58.820794 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.820763 5420 generic.go:356] "Generic (PLEG): container finished" podID="77786272-402f-4862-96f1-872cfcd090a9" containerID="8a51d14b809acc29ac976f946373e1a1ab795b411d451d8ca77549f17f763176" exitCode=0 Nov 02 20:42:58.820927 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.820839 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" event={"ID":"77786272-402f-4862-96f1-872cfcd090a9","Type":"ContainerDied","Data":"8a51d14b809acc29ac976f946373e1a1ab795b411d451d8ca77549f17f763176"} Nov 02 20:42:58.822814 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.822794 5420 generic.go:356] "Generic (PLEG): container finished" podID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerID="c8396d34d13cb40a105e3fdfb66b76c7093c862bbb3b5e38a703c0e980c94086" exitCode=0 Nov 02 20:42:58.822933 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.822828 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" event={"ID":"a3fb4a0b-a907-4170-8367-fb9fe4479d0c","Type":"ContainerDied","Data":"c8396d34d13cb40a105e3fdfb66b76c7093c862bbb3b5e38a703c0e980c94086"} Nov 02 20:42:58.824297 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.824276 5420 generic.go:356] "Generic (PLEG): container finished" podID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerID="656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5" exitCode=0 Nov 02 20:42:58.824466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.824448 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" event={"ID":"5974eac6-37e4-4cdd-bb61-b1946397079a","Type":"ContainerDied","Data":"656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5"} Nov 02 20:42:58.824707 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.824491 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" Nov 02 20:42:58.824804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.824726 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9" event={"ID":"5974eac6-37e4-4cdd-bb61-b1946397079a","Type":"ContainerDied","Data":"7e552b1694016e8adb98303191709282c3e872661a07331adba81d4bf9460742"} Nov 02 20:42:58.824804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.824781 5420 scope.go:122] "RemoveContainer" containerID="656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5" Nov 02 20:42:58.838171 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.838007 5420 scope.go:122] "RemoveContainer" containerID="656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5" Nov 02 20:42:58.838441 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:42:58.838419 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5\": container with ID starting with 656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5 not found: ID does not exist" containerID="656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5" Nov 02 20:42:58.838587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.838565 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5"} err="failed to get container status \"656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5\": rpc error: code = NotFound desc = could not find container \"656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5\": container with ID starting with 656540f2708fd073752801875019fef1add52cf0a9712acfe12c8f187eb47be5 not found: ID does not exist" Nov 02 20:42:58.925497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.925475 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:42:58.927832 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:58.927816 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:42:59.008424 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008335 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-credential-socket\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.008591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008441 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5974eac6-37e4-4cdd-bb61-b1946397079a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5974eac6-37e4-4cdd-bb61-b1946397079a-istiod-ca-cert\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.008591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008474 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-certs\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.008760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008593 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-token\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-token\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.008760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008654 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-kube-api-access-zvcc9\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-kube-api-access-zvcc9\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.008760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008731 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-socket\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.008923 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008818 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-podinfo\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.008923 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008867 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-envoy\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.009024 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.008981 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-data\") pod \"5974eac6-37e4-4cdd-bb61-b1946397079a\" (UID: \"5974eac6-37e4-4cdd-bb61-b1946397079a\") " Nov 02 20:42:59.010075 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.010038 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-data" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.010284 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.010263 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-credential-socket" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.010642 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.010622 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5974eac6-37e4-4cdd-bb61-b1946397079a-istiod-ca-cert" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:59.010937 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.010915 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-certs" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.011569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.011543 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-socket" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.013407 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.013383 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-token" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:59.013884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.013864 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-podinfo" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:42:59.014318 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.014296 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-kube-api-access-zvcc9" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "kube-api-access-zvcc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:59.014407 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.014350 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-envoy" pod "5974eac6-37e4-4cdd-bb61-b1946397079a" (UID: "5974eac6-37e4-4cdd-bb61-b1946397079a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.111309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111279 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-socket\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.111309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111319 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77786272-402f-4862-96f1-872cfcd090a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77786272-402f-4862-96f1-872cfcd090a9-istio-podinfo\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.111558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111350 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-credential-socket\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.111558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111373 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-socket\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.111558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111428 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-certs\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.111558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111472 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-credential-socket\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.111558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111542 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77786272-402f-4862-96f1-872cfcd090a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77786272-402f-4862-96f1-872cfcd090a9-istiod-ca-cert\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.111911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111626 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istiod-ca-cert\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.111911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111663 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-certs\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.111911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111746 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-data\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.111911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111778 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-kube-api-access-vgzxn\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-kube-api-access-vgzxn\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.111911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111882 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-envoy\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.112157 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111936 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-podinfo\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.112157 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.111964 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-data\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.112157 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.112020 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-envoy\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.112157 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.112056 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-token\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.112157 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.112083 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-istio-token\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-istio-token\") pod \"77786272-402f-4862-96f1-872cfcd090a9\" (UID: \"77786272-402f-4862-96f1-872cfcd090a9\") " Nov 02 20:42:59.112391 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.112170 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-kube-api-access-c72pl\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-kube-api-access-c72pl\") pod \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\" (UID: \"a3fb4a0b-a907-4170-8367-fb9fe4479d0c\") " Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.112534 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-certs" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113181 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113210 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvcc9\" (UniqueName: \"kubernetes.io/projected/5974eac6-37e4-4cdd-bb61-b1946397079a-kube-api-access-zvcc9\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113243 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113259 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113273 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113286 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113299 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113327 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5974eac6-37e4-4cdd-bb61-b1946397079a-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113340 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5974eac6-37e4-4cdd-bb61-b1946397079a-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.113618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113352 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.114323 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.113663 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-socket" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.119328 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.115927 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-data" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.119328 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.116163 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-credential-socket" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.119328 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.118186 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77786272-402f-4862-96f1-872cfcd090a9-istiod-ca-cert" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:59.119328 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.118512 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-credential-socket" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.119328 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.118739 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-socket" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.119328 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.119023 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istiod-ca-cert" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:42:59.121405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.121357 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-data" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.121588 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.121556 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-certs" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.135226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.135128 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-kube-api-access-vgzxn" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "kube-api-access-vgzxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:59.135226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.135164 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-token" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:59.135226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.135193 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-kube-api-access-c72pl" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "kube-api-access-c72pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:59.136164 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.136129 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-envoy" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.136399 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.136369 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-envoy" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:42:59.136588 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.136570 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-podinfo" pod "a3fb4a0b-a907-4170-8367-fb9fe4479d0c" (UID: "a3fb4a0b-a907-4170-8367-fb9fe4479d0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:42:59.139793 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.139564 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-istio-token" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:42:59.139909 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.139887 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77786272-402f-4862-96f1-872cfcd090a9-istio-podinfo" pod "77786272-402f-4862-96f1-872cfcd090a9" (UID: "77786272-402f-4862-96f1-872cfcd090a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:42:59.223801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223765 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.223801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223804 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223815 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223826 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72pl\" (UniqueName: \"kubernetes.io/projected/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-kube-api-access-c72pl\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223836 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223844 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77786272-402f-4862-96f1-872cfcd090a9-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223853 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223861 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223870 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223880 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223889 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77786272-402f-4862-96f1-872cfcd090a9-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223897 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223905 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77786272-402f-4862-96f1-872cfcd090a9-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223914 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgzxn\" (UniqueName: \"kubernetes.io/projected/77786272-402f-4862-96f1-872cfcd090a9-kube-api-access-vgzxn\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223934 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223945 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.224536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.223956 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fb4a0b-a907-4170-8367-fb9fe4479d0c-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:42:59.402556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.402509 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9"] Nov 02 20:42:59.406767 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.406740 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-r4dw9"] Nov 02 20:42:59.801461 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.801203 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" path="/var/lib/kubelet/pods/5974eac6-37e4-4cdd-bb61-b1946397079a/volumes" Nov 02 20:42:59.832422 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.832402 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" Nov 02 20:42:59.832824 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.832440 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2" event={"ID":"77786272-402f-4862-96f1-872cfcd090a9","Type":"ContainerDied","Data":"f5bd5a231c2e15f0ba46d5b8d13c39b8a3ae7d70d1415fe32e9613e08010900c"} Nov 02 20:42:59.832824 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.832509 5420 scope.go:122] "RemoveContainer" containerID="8a51d14b809acc29ac976f946373e1a1ab795b411d451d8ca77549f17f763176" Nov 02 20:42:59.834005 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.833984 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" Nov 02 20:42:59.834005 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.833999 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq" event={"ID":"a3fb4a0b-a907-4170-8367-fb9fe4479d0c","Type":"ContainerDied","Data":"34ab8ed7a442b9ca8015e6d5f9894fdce13864182ae279f116c31db72e545b3f"} Nov 02 20:42:59.843127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.843105 5420 scope.go:122] "RemoveContainer" containerID="c8396d34d13cb40a105e3fdfb66b76c7093c862bbb3b5e38a703c0e980c94086" Nov 02 20:42:59.996908 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:42:59.996867 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq"] Nov 02 20:43:00.000939 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:00.000906 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-lxgbq"] Nov 02 20:43:00.105208 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:00.105131 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2"] Nov 02 20:43:00.108922 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:00.108897 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-b8cq2"] Nov 02 20:43:00.585379 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:00.585346 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg"] Nov 02 20:43:00.585608 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:00.585574 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" podUID="0641a1c3-5451-41d1-8d94-d0c48ab8b117" containerName="maas-api" containerID="cri-o://b9830f2dbadb641e73b67666c449b9e64f664da42d7f3104f704fe3cb4989867" gracePeriod=30 Nov 02 20:43:00.929116 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:00.928979 5420 generic.go:356] "Generic (PLEG): container finished" podID="0641a1c3-5451-41d1-8d94-d0c48ab8b117" containerID="b9830f2dbadb641e73b67666c449b9e64f664da42d7f3104f704fe3cb4989867" exitCode=0 Nov 02 20:43:00.929418 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:00.929136 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" event={"ID":"0641a1c3-5451-41d1-8d94-d0c48ab8b117","Type":"ContainerDied","Data":"b9830f2dbadb641e73b67666c449b9e64f664da42d7f3104f704fe3cb4989867"} Nov 02 20:43:01.045278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.045244 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:43:01.154382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.154353 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-metrics-certs\") pod \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " Nov 02 20:43:01.154567 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.154398 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0641a1c3-5451-41d1-8d94-d0c48ab8b117-kube-api-access-rvjv6\" (UniqueName: \"kubernetes.io/projected/0641a1c3-5451-41d1-8d94-d0c48ab8b117-kube-api-access-rvjv6\") pod \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " Nov 02 20:43:01.154628 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.154606 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls\") pod \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " Nov 02 20:43:01.154815 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.154665 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0641a1c3-5451-41d1-8d94-d0c48ab8b117-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641a1c3-5451-41d1-8d94-d0c48ab8b117-openshift-service-ca\") pod \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\" (UID: \"0641a1c3-5451-41d1-8d94-d0c48ab8b117\") " Nov 02 20:43:01.155063 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.155035 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0641a1c3-5451-41d1-8d94-d0c48ab8b117-openshift-service-ca" pod "0641a1c3-5451-41d1-8d94-d0c48ab8b117" (UID: "0641a1c3-5451-41d1-8d94-d0c48ab8b117"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:01.155459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.155437 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0641a1c3-5451-41d1-8d94-d0c48ab8b117-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:01.193925 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.192018 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-metrics-certs" pod "0641a1c3-5451-41d1-8d94-d0c48ab8b117" (UID: "0641a1c3-5451-41d1-8d94-d0c48ab8b117"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:01.193925 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.192128 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls" pod "0641a1c3-5451-41d1-8d94-d0c48ab8b117" (UID: "0641a1c3-5451-41d1-8d94-d0c48ab8b117"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:01.193925 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.192840 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0641a1c3-5451-41d1-8d94-d0c48ab8b117-kube-api-access-rvjv6" pod "0641a1c3-5451-41d1-8d94-d0c48ab8b117" (UID: "0641a1c3-5451-41d1-8d94-d0c48ab8b117"). InnerVolumeSpecName "kube-api-access-rvjv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:01.266691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.258081 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:01.266691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.258114 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjv6\" (UniqueName: \"kubernetes.io/projected/0641a1c3-5451-41d1-8d94-d0c48ab8b117-kube-api-access-rvjv6\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:01.266691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.258129 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0641a1c3-5451-41d1-8d94-d0c48ab8b117-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:01.827955 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.827916 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77786272-402f-4862-96f1-872cfcd090a9" path="/var/lib/kubelet/pods/77786272-402f-4862-96f1-872cfcd090a9/volumes" Nov 02 20:43:01.828620 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.828599 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" path="/var/lib/kubelet/pods/a3fb4a0b-a907-4170-8367-fb9fe4479d0c/volumes" Nov 02 20:43:01.946845 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.946799 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" Nov 02 20:43:01.947334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.946821 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg" event={"ID":"0641a1c3-5451-41d1-8d94-d0c48ab8b117","Type":"ContainerDied","Data":"ae90f5760d1481efbe6261d1596febb735a43a023480ce968861d0618ad3b878"} Nov 02 20:43:01.947474 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:01.947460 5420 scope.go:122] "RemoveContainer" containerID="b9830f2dbadb641e73b67666c449b9e64f664da42d7f3104f704fe3cb4989867" Nov 02 20:43:02.103632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.103549 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg"] Nov 02 20:43:02.221693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.220859 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-23d588ad-5d5bf7888d-sbwbg"] Nov 02 20:43:02.354746 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.354052 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn"] Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382633 5420 cpu_manager.go:414] "containerMap: removing container" podUID="0641a1c3-5451-41d1-8d94-d0c48ab8b117" containerName="maas-api" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382659 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0641a1c3-5451-41d1-8d94-d0c48ab8b117" containerName="maas-api" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382736 5420 cpu_manager.go:414] "containerMap: removing container" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382745 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382785 5420 cpu_manager.go:414] "containerMap: removing container" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382796 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382809 5420 cpu_manager.go:414] "containerMap: removing container" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382816 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.382988 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0641a1c3-5451-41d1-8d94-d0c48ab8b117" containerName="maas-api" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.383010 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="77786272-402f-4862-96f1-872cfcd090a9" containerName="istio-proxy" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.383024 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="5974eac6-37e4-4cdd-bb61-b1946397079a" containerName="istio-proxy" Nov 02 20:43:02.384703 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.383035 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3fb4a0b-a907-4170-8367-fb9fe4479d0c" containerName="istio-proxy" Nov 02 20:43:02.390696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.389204 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.478593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.478564 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-0ea576-serving-cert\"" Nov 02 20:43:02.498704 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.481242 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3595205a-4595-486b-996d-38dbda6359c1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.498704 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.494178 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.498704 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.494279 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.498704 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.494309 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhnk8\" (UniqueName: \"kubernetes.io/projected/3595205a-4595-486b-996d-38dbda6359c1-kube-api-access-xhnk8\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.508937 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.502172 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn"] Nov 02 20:43:02.596012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.595976 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.596176 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:02.596107 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-serving-cert: secret "maas-api-e2e-shared-a-w0-0ea576-serving-cert" not found Nov 02 20:43:02.596176 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.596103 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.596260 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:02.596196 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls podName:3595205a-4595-486b-996d-38dbda6359c1 nodeName:}" failed. No retries permitted until 2026-11-02 20:43:03.096145438 +0000 UTC m=+1655.460756634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls") pod "maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" (UID: "3595205a-4595-486b-996d-38dbda6359c1") : secret "maas-api-e2e-shared-a-w0-0ea576-serving-cert" not found Nov 02 20:43:02.596260 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.596223 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhnk8\" (UniqueName: \"kubernetes.io/projected/3595205a-4595-486b-996d-38dbda6359c1-kube-api-access-xhnk8\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.596523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.596503 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3595205a-4595-486b-996d-38dbda6359c1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.597186 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.597158 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3595205a-4595-486b-996d-38dbda6359c1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.599012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.598993 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.684196 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:02.682039 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhnk8\" (UniqueName: \"kubernetes.io/projected/3595205a-4595-486b-996d-38dbda6359c1-kube-api-access-xhnk8\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:02.724722 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:02.724323 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:43:03.111830 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.111799 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:03.114392 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.114366 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:03.362009 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.361927 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659"] Nov 02 20:43:03.367337 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.367315 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.439696 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.411312 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.497966 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-4166b0-openshift-default-dockercfg-hg8fh\"" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512260 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-socket\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512342 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-credential-socket\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512386 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-data\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512441 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-envoy\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512470 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxw69\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-kube-api-access-cxw69\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512542 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-certs\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512755 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37050465-6348-48cb-90bd-36272f806434-istiod-ca-cert\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512869 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-istio-token\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.555768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.512956 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37050465-6348-48cb-90bd-36272f806434-istio-podinfo\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.577535 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.577495 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659"] Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.616580 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37050465-6348-48cb-90bd-36272f806434-istio-podinfo\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.616719 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-socket\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.616802 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-credential-socket\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.616841 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-data\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.616905 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-envoy\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.616934 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxw69\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-kube-api-access-cxw69\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.617000 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-certs\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.617177 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37050465-6348-48cb-90bd-36272f806434-istiod-ca-cert\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.617268 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-istio-token\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.619263 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-credential-socket\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.619327 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37050465-6348-48cb-90bd-36272f806434-istiod-ca-cert\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.619502 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-socket\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.619550 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-data\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.644090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.619641 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-certs\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.664394 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.648880 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37050465-6348-48cb-90bd-36272f806434-istio-podinfo\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.664394 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.656973 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-envoy\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.729767 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.726361 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxw69\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-kube-api-access-cxw69\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.744415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.735967 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-istio-token\") pod \"e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:03.798081 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:03.797887 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0641a1c3-5451-41d1-8d94-d0c48ab8b117" path="/var/lib/kubelet/pods/0641a1c3-5451-41d1-8d94-d0c48ab8b117/volumes" Nov 02 20:43:04.015746 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:04.011955 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:04.102297 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:04.102199 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn"] Nov 02 20:43:04.530929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:04.530896 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659"] Nov 02 20:43:04.531964 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:43:04.531933 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37050465_6348_48cb_90bd_36272f806434.slice/crio-81fd5eb35d593acb09a0e9e2f05ad43da8f5fa4d4631a8633f43ee6246cc5d49 WatchSource:0}: Error finding container 81fd5eb35d593acb09a0e9e2f05ad43da8f5fa4d4631a8633f43ee6246cc5d49: Status 404 returned error can't find the container with id 81fd5eb35d593acb09a0e9e2f05ad43da8f5fa4d4631a8633f43ee6246cc5d49 Nov 02 20:43:04.535026 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:04.534986 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:04.535103 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:04.535078 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:04.535153 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:04.535123 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:05.064931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:05.064837 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" event={"ID":"37050465-6348-48cb-90bd-36272f806434","Type":"ContainerStarted","Data":"e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d"} Nov 02 20:43:05.064931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:05.064900 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" event={"ID":"37050465-6348-48cb-90bd-36272f806434","Type":"ContainerStarted","Data":"81fd5eb35d593acb09a0e9e2f05ad43da8f5fa4d4631a8633f43ee6246cc5d49"} Nov 02 20:43:05.068224 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:05.068189 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" event={"ID":"3595205a-4595-486b-996d-38dbda6359c1","Type":"ContainerStarted","Data":"dfc3daaa94f21401fef1bb169031c6e0fad481a187ba43e3536a18d21285e3a8"} Nov 02 20:43:05.178825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:05.178763 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" podStartSLOduration=2.178743966 podStartE2EDuration="2.178743966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:05.17694837 +0000 UTC m=+1657.541559575" watchObservedRunningTime="2026-11-02 20:43:05.178743966 +0000 UTC m=+1657.543355171" Nov 02 20:43:06.012996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:06.012960 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:06.023179 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:06.023149 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:06.082577 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:06.082543 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" event={"ID":"3595205a-4595-486b-996d-38dbda6359c1","Type":"ContainerStarted","Data":"5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c"} Nov 02 20:43:06.082804 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:06.082626 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:06.083310 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:06.082960 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:06.084101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:06.084078 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:43:06.291624 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:06.291422 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" podStartSLOduration=3.349932817 podStartE2EDuration="4.291405313s" totalImagesPullingTime="941.472496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:06.290753319 +0000 UTC m=+1658.655364523" watchObservedRunningTime="2026-11-02 20:43:06.291405313 +0000 UTC m=+1658.656016520" Nov 02 20:43:07.094906 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:07.094869 5420 generic.go:356] "Generic (PLEG): container finished" podID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerID="c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea" exitCode=0 Nov 02 20:43:07.095364 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:07.094949 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" event={"ID":"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d","Type":"ContainerDied","Data":"c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea"} Nov 02 20:43:07.095741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:07.095719 5420 scope.go:122] "RemoveContainer" containerID="c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea" Nov 02 20:43:07.097041 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:07.097012 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:07.097130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:07.097075 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:07.097130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:07.097113 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:08.105653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:08.105614 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" event={"ID":"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d","Type":"ContainerStarted","Data":"64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c"} Nov 02 20:43:08.853455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:08.853421 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:43:08.854987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:08.854957 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:43:08.855136 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:08.855053 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:43:09.852903 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:09.852870 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:43:09.853280 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:09.852930 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:43:10.852734 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:10.852699 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:43:10.852938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:10.852751 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:43:11.853598 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:11.853562 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:43:11.853995 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:11.853625 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:43:12.101231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:12.101203 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:43:12.853462 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:12.853426 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 02 20:43:12.853625 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:12.853488 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 02 20:43:12.892154 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:12.892108 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:43:13.858446 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:13.858413 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:43:13.866140 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:13.866114 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:43:14.555946 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:14.555909 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn"] Nov 02 20:43:22.169579 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:22.169547 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2"] Nov 02 20:43:27.186782 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:27.179730 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" podUID="dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" containerName="payload-processing" containerID="cri-o://26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c" gracePeriod=25 Nov 02 20:43:29.589912 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.589869 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" containerID="cri-o://0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9" gracePeriod=45 Nov 02 20:43:29.889559 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.889535 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:43:29.940568 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.940529 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-tmp-dir\") pod \"e9e1d983-6558-405b-ad3a-58f3121815e6\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " Nov 02 20:43:29.940768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.940583 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e1d983-6558-405b-ad3a-58f3121815e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e1d983-6558-405b-ad3a-58f3121815e6-tls-certs\") pod \"e9e1d983-6558-405b-ad3a-58f3121815e6\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " Nov 02 20:43:29.940768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.940651 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e1d983-6558-405b-ad3a-58f3121815e6-kube-api-access-zpdfb\" (UniqueName: \"kubernetes.io/projected/e9e1d983-6558-405b-ad3a-58f3121815e6-kube-api-access-zpdfb\") pod \"e9e1d983-6558-405b-ad3a-58f3121815e6\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " Nov 02 20:43:29.940903 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.940799 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-tmp-dir" pod "e9e1d983-6558-405b-ad3a-58f3121815e6" (UID: "e9e1d983-6558-405b-ad3a-58f3121815e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:29.940903 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.940889 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-dshm\") pod \"e9e1d983-6558-405b-ad3a-58f3121815e6\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " Nov 02 20:43:29.941338 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.941315 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-model-cache\") pod \"e9e1d983-6558-405b-ad3a-58f3121815e6\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " Nov 02 20:43:29.941455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.941377 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-home\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-home\") pod \"e9e1d983-6558-405b-ad3a-58f3121815e6\" (UID: \"e9e1d983-6558-405b-ad3a-58f3121815e6\") " Nov 02 20:43:29.941754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.941665 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-model-cache" pod "e9e1d983-6558-405b-ad3a-58f3121815e6" (UID: "e9e1d983-6558-405b-ad3a-58f3121815e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:29.941898 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.941876 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-home" pod "e9e1d983-6558-405b-ad3a-58f3121815e6" (UID: "e9e1d983-6558-405b-ad3a-58f3121815e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:29.942234 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.942216 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:29.942334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.942238 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:29.942334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.942251 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:29.943013 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.942991 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-dshm" pod "e9e1d983-6558-405b-ad3a-58f3121815e6" (UID: "e9e1d983-6558-405b-ad3a-58f3121815e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:29.943178 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.943156 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e1d983-6558-405b-ad3a-58f3121815e6-tls-certs" pod "e9e1d983-6558-405b-ad3a-58f3121815e6" (UID: "e9e1d983-6558-405b-ad3a-58f3121815e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:43:29.943225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:29.943159 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e1d983-6558-405b-ad3a-58f3121815e6-kube-api-access-zpdfb" pod "e9e1d983-6558-405b-ad3a-58f3121815e6" (UID: "e9e1d983-6558-405b-ad3a-58f3121815e6"). InnerVolumeSpecName "kube-api-access-zpdfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:30.043976 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.043947 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e1d983-6558-405b-ad3a-58f3121815e6-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.043976 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.043978 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e1d983-6558-405b-ad3a-58f3121815e6-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.044200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.043991 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpdfb\" (UniqueName: \"kubernetes.io/projected/e9e1d983-6558-405b-ad3a-58f3121815e6-kube-api-access-zpdfb\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:30.422861 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.422830 5420 generic.go:356] "Generic (PLEG): container finished" podID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerID="0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9" exitCode=0 Nov 02 20:43:30.423037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.422882 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" Nov 02 20:43:30.423037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.422913 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" event={"ID":"e9e1d983-6558-405b-ad3a-58f3121815e6","Type":"ContainerDied","Data":"0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9"} Nov 02 20:43:30.423037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.422967 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn" event={"ID":"e9e1d983-6558-405b-ad3a-58f3121815e6","Type":"ContainerDied","Data":"4dfaf53708f8a8761b77e50c89dad9f11e5a497d68122674d9b014409c8cc1d1"} Nov 02 20:43:30.423037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.422989 5420 scope.go:122] "RemoveContainer" containerID="0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9" Nov 02 20:43:30.444584 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.437385 5420 scope.go:122] "RemoveContainer" containerID="0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9" Nov 02 20:43:30.444584 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:30.438171 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9\": container with ID starting with 0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9 not found: ID does not exist" containerID="0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9" Nov 02 20:43:30.444584 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.438223 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9"} err="failed to get container status \"0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9\": rpc error: code = NotFound desc = could not find container \"0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9\": container with ID starting with 0810a4203acc2a6017a47654b881f298575c6242ba2eeb7c7eb639bcf73988a9 not found: ID does not exist" Nov 02 20:43:30.582241 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.582206 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn"] Nov 02 20:43:30.589866 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:30.589833 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-694cee/distinct-w2-694cee-kserve-59f7d8d9b9-sqtfn"] Nov 02 20:43:31.790632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:31.790593 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" path="/var/lib/kubelet/pods/e9e1d983-6558-405b-ad3a-58f3121815e6/volumes" Nov 02 20:43:32.667079 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:32.667040 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:43:33.450521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:33.450488 5420 generic.go:356] "Generic (PLEG): container finished" podID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerID="a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc" exitCode=0 Nov 02 20:43:33.450963 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:33.450561 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" event={"ID":"eaa56fb7-abaa-4169-9c0a-319feee7dc86","Type":"ContainerDied","Data":"a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc"} Nov 02 20:43:33.451006 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:33.450972 5420 scope.go:122] "RemoveContainer" containerID="a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc" Nov 02 20:43:33.452439 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:33.452414 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:33.452493 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:33.452480 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:33.452530 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:33.452513 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:43:34.269693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:34.264757 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:43:34.461223 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:34.461180 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" event={"ID":"eaa56fb7-abaa-4169-9c0a-319feee7dc86","Type":"ContainerStarted","Data":"5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651"} Nov 02 20:43:34.521542 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:34.521436 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" podUID="dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:34.771023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:34.770962 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:43:34.786560 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:34.786462 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:43:34.786882 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:34.786851 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:43:35.794747 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:35.783290 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:43:35.794747 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:35.783346 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:43:36.780610 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:36.780572 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:43:36.780924 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:36.780896 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:43:37.435822 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:37.435788 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz"] Nov 02 20:43:37.831697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:37.831266 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:43:37.831697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:37.831338 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:43:38.419153 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.419103 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj"] Nov 02 20:43:38.419397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.419358 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" containerID="cri-o://3c9af6f70ec9bef9bd8b528a82938230158e15b0eb8ee4f508356be71b01f9f1" gracePeriod=30 Nov 02 20:43:38.422970 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.422940 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx"] Nov 02 20:43:38.424174 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.423139 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" containerID="cri-o://3b8a0ec1db1b1e898f2a5a4227cc951786915e7887ba5d2fc547f3b709e1a3ce" gracePeriod=30 Nov 02 20:43:38.495078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.495038 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs"] Nov 02 20:43:38.495858 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.495807 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" containerID="cri-o://24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631" gracePeriod=30 Nov 02 20:43:38.504212 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.504170 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth"] Nov 02 20:43:38.504565 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.504518 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" containerID="cri-o://70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2" gracePeriod=30 Nov 02 20:43:38.787167 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.787081 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 02 20:43:38.787408 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:38.787380 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 02 20:43:39.846204 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:39.846171 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:43:39.846951 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:39.846931 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:43:39.847821 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:39.847799 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:43:40.972939 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:40.972908 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:43:40.973288 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:40.972973 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:43:42.436332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.436279 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" podUID="52f68223-bd3d-496b-a498-be15cc8bdec7" containerName="payload-pre-processing" containerID="cri-o://a67eae9c0b09be656ec4b27a7d8aa8f55142d2b6ca3c4bd376b48d8835055c70" gracePeriod=25 Nov 02 20:43:42.442602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.442578 5420 patch_prober.go:29] interesting pod/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:43:42.442752 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.442646 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:43:42.557522 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.557499 5420 generic.go:356] "Generic (PLEG): container finished" podID="52f68223-bd3d-496b-a498-be15cc8bdec7" containerID="a67eae9c0b09be656ec4b27a7d8aa8f55142d2b6ca3c4bd376b48d8835055c70" exitCode=0 Nov 02 20:43:42.557634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.557575 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" event={"ID":"52f68223-bd3d-496b-a498-be15cc8bdec7","Type":"ContainerDied","Data":"a67eae9c0b09be656ec4b27a7d8aa8f55142d2b6ca3c4bd376b48d8835055c70"} Nov 02 20:43:42.723512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.723482 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:43:42.837784 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.837752 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f68223-bd3d-496b-a498-be15cc8bdec7-kube-api-access-t58dh\" (UniqueName: \"kubernetes.io/projected/52f68223-bd3d-496b-a498-be15cc8bdec7-kube-api-access-t58dh\") pod \"52f68223-bd3d-496b-a498-be15cc8bdec7\" (UID: \"52f68223-bd3d-496b-a498-be15cc8bdec7\") " Nov 02 20:43:42.837937 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.837797 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f68223-bd3d-496b-a498-be15cc8bdec7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f68223-bd3d-496b-a498-be15cc8bdec7-plugins-config-volume\") pod \"52f68223-bd3d-496b-a498-be15cc8bdec7\" (UID: \"52f68223-bd3d-496b-a498-be15cc8bdec7\") " Nov 02 20:43:42.838254 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.838229 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f68223-bd3d-496b-a498-be15cc8bdec7-plugins-config-volume" pod "52f68223-bd3d-496b-a498-be15cc8bdec7" (UID: "52f68223-bd3d-496b-a498-be15cc8bdec7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:42.838604 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.838585 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f68223-bd3d-496b-a498-be15cc8bdec7-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:42.839865 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.839838 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f68223-bd3d-496b-a498-be15cc8bdec7-kube-api-access-t58dh" pod "52f68223-bd3d-496b-a498-be15cc8bdec7" (UID: "52f68223-bd3d-496b-a498-be15cc8bdec7"). InnerVolumeSpecName "kube-api-access-t58dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:42.940724 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:42.940700 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t58dh\" (UniqueName: \"kubernetes.io/projected/52f68223-bd3d-496b-a498-be15cc8bdec7-kube-api-access-t58dh\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.568326 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.568299 5420 generic.go:356] "Generic (PLEG): container finished" podID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerID="3b8a0ec1db1b1e898f2a5a4227cc951786915e7887ba5d2fc547f3b709e1a3ce" exitCode=0 Nov 02 20:43:43.568689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.568380 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" event={"ID":"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1","Type":"ContainerDied","Data":"3b8a0ec1db1b1e898f2a5a4227cc951786915e7887ba5d2fc547f3b709e1a3ce"} Nov 02 20:43:43.570150 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.570130 5420 generic.go:356] "Generic (PLEG): container finished" podID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerID="3c9af6f70ec9bef9bd8b528a82938230158e15b0eb8ee4f508356be71b01f9f1" exitCode=0 Nov 02 20:43:43.570228 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.570199 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" event={"ID":"54d935d8-83fb-458c-a4ef-f0a0baf669be","Type":"ContainerDied","Data":"3c9af6f70ec9bef9bd8b528a82938230158e15b0eb8ee4f508356be71b01f9f1"} Nov 02 20:43:43.571530 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.571504 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" event={"ID":"52f68223-bd3d-496b-a498-be15cc8bdec7","Type":"ContainerDied","Data":"91d9d025a5600eae8b91e3e94b6fdcc1508bd67957719db8050629f31b3891f0"} Nov 02 20:43:43.571883 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.571543 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz" Nov 02 20:43:43.571883 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.571555 5420 scope.go:122] "RemoveContainer" containerID="a67eae9c0b09be656ec4b27a7d8aa8f55142d2b6ca3c4bd376b48d8835055c70" Nov 02 20:43:43.822921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.822298 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz"] Nov 02 20:43:43.834054 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.834028 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3d8f2f-98ddb4f99-6jkmz"] Nov 02 20:43:43.842911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.842891 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:43:43.861481 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.861457 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-data\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.861814 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.861793 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-kube-api-access-4h2vv\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-kube-api-access-4h2vv\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.861973 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.861955 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-envoy\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.862113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.862098 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d935d8-83fb-458c-a4ef-f0a0baf669be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d935d8-83fb-458c-a4ef-f0a0baf669be-istiod-ca-cert\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.862298 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.862282 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-token\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-token\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.862457 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.862426 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-credential-socket\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.862645 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.862626 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-socket\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.862797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.862779 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-podinfo\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.863016 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.862999 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-certs\") pod \"54d935d8-83fb-458c-a4ef-f0a0baf669be\" (UID: \"54d935d8-83fb-458c-a4ef-f0a0baf669be\") " Nov 02 20:43:43.864875 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.862300 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-data" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.864984 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.863978 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-certs" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.865078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.864248 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d935d8-83fb-458c-a4ef-f0a0baf669be-istiod-ca-cert" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:43.865163 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.864852 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-credential-socket" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.866514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.866487 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-socket" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.866963 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.866923 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-envoy" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.867962 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.867928 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-podinfo" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:43.871948 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.871925 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-token" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:43.872081 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.872050 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-kube-api-access-4h2vv" pod "54d935d8-83fb-458c-a4ef-f0a0baf669be" (UID: "54d935d8-83fb-458c-a4ef-f0a0baf669be"). InnerVolumeSpecName "kube-api-access-4h2vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:43.891989 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.891969 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:43:43.964357 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.964328 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-token\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-token\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.964513 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.964390 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istiod-ca-cert\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.964580 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.964526 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-podinfo\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.965735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.964639 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-certs\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.965735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.965184 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istiod-ca-cert" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:43.965735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.965256 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-certs" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.965735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.965505 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-socket\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.965735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.965558 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-credential-socket\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.965735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.965696 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-kube-api-access-szh54\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-kube-api-access-szh54\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.965735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.965733 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-envoy\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.966138 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.965775 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-data\") pod \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\" (UID: \"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1\") " Nov 02 20:43:43.966138 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.965963 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-credential-socket" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.966246 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966193 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-socket" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.966829 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966807 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966835 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h2vv\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-kube-api-access-4h2vv\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966851 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966864 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d935d8-83fb-458c-a4ef-f0a0baf669be-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966878 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966891 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966905 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966920 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966933 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966947 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966960 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d935d8-83fb-458c-a4ef-f0a0baf669be-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966973 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.966986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.966986 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:43.967566 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.967185 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-data" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.968858 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.968833 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-kube-api-access-szh54" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "kube-api-access-szh54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:43.970575 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.970548 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-envoy" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:43.974137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.971186 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-podinfo" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:43.974137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.971423 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:43:43.974137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.971833 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-token" pod "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" (UID: "cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:43.978847 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:43.978656 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:43:44.067598 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067577 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-credential-socket\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.067759 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067614 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-certs\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.067759 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067713 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-podinfo\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.067759 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067755 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-istio-token\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-istio-token\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.067930 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067800 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-certs\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.067930 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067850 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-envoy\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.067930 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067851 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-credential-socket" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.067930 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067892 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-envoy\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.068169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067944 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-certs" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.068169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.067954 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/636700c4-2720-4f52-8043-d15cead9abf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636700c4-2720-4f52-8043-d15cead9abf4-istio-podinfo\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.068169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068020 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-socket\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.068169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068150 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-credential-socket\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.068374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068202 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-data\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.068374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068230 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636700c4-2720-4f52-8043-d15cead9abf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636700c4-2720-4f52-8043-d15cead9abf4-istiod-ca-cert\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.068374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068277 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-socket\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.068374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068299 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-socket" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.068374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068317 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-data\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.068374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068352 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-kube-api-access-8rzns\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-kube-api-access-8rzns\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.068658 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068385 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istiod-ca-cert\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.068658 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068419 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-token\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-token\") pod \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\" (UID: \"fc9eb207-3df4-4ba5-acda-4041ef5bc080\") " Nov 02 20:43:44.068658 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068463 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-certs" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.068658 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068465 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-kube-api-access-qb7lt\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-kube-api-access-qb7lt\") pod \"636700c4-2720-4f52-8043-d15cead9abf4\" (UID: \"636700c4-2720-4f52-8043-d15cead9abf4\") " Nov 02 20:43:44.069001 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.068969 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-data" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.069549 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636700c4-2720-4f52-8043-d15cead9abf4-istiod-ca-cert" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.069845 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-data" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070054 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-socket" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070108 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istiod-ca-cert" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070310 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-credential-socket" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070738 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070756 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070770 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070783 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070797 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070810 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070823 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636700c4-2720-4f52-8043-d15cead9abf4-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070837 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070850 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070863 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szh54\" (UniqueName: \"kubernetes.io/projected/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-kube-api-access-szh54\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070877 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070890 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070903 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070915 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.070930 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.072621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.072521 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-token" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.073952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.072809 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/636700c4-2720-4f52-8043-d15cead9abf4-istio-podinfo" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:44.073952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.072815 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-kube-api-access-qb7lt" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "kube-api-access-qb7lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.073952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.072865 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-envoy" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.073952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.072887 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-podinfo" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:44.073952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.073921 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-kube-api-access-8rzns" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "kube-api-access-8rzns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.074356 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.074333 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-istio-token" pod "636700c4-2720-4f52-8043-d15cead9abf4" (UID: "636700c4-2720-4f52-8043-d15cead9abf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:44.076181 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.076157 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-envoy" pod "fc9eb207-3df4-4ba5-acda-4041ef5bc080" (UID: "fc9eb207-3df4-4ba5-acda-4041ef5bc080"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:44.171952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.171916 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636700c4-2720-4f52-8043-d15cead9abf4-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.171952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.171944 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.171952 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.171959 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636700c4-2720-4f52-8043-d15cead9abf4-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.172184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.171972 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rzns\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-kube-api-access-8rzns\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.172184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.171983 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.172184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.171994 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb7lt\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-kube-api-access-qb7lt\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.172184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.172005 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc9eb207-3df4-4ba5-acda-4041ef5bc080-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.172184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.172018 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636700c4-2720-4f52-8043-d15cead9abf4-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:44.520726 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.520693 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" podUID="dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:43:44.586594 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.586561 5420 generic.go:356] "Generic (PLEG): container finished" podID="636700c4-2720-4f52-8043-d15cead9abf4" containerID="70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2" exitCode=0 Nov 02 20:43:44.586594 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.586603 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" Nov 02 20:43:44.587125 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.586640 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" event={"ID":"636700c4-2720-4f52-8043-d15cead9abf4","Type":"ContainerDied","Data":"70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2"} Nov 02 20:43:44.587125 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.586735 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth" event={"ID":"636700c4-2720-4f52-8043-d15cead9abf4","Type":"ContainerDied","Data":"cf96df2ae22029bdddd3779248fc9d5f5fa4a07db021c8e6f805b4789b5352de"} Nov 02 20:43:44.587125 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.586763 5420 scope.go:122] "RemoveContainer" containerID="70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2" Nov 02 20:43:44.588860 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.588837 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" event={"ID":"cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1","Type":"ContainerDied","Data":"2cd3bd5623b32bbed3311ec2189a78a537f06c2db9662cd807f49c0687502bfc"} Nov 02 20:43:44.589139 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.589117 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx" Nov 02 20:43:44.591017 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.590988 5420 generic.go:356] "Generic (PLEG): container finished" podID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerID="24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631" exitCode=0 Nov 02 20:43:44.591171 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.591155 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" event={"ID":"fc9eb207-3df4-4ba5-acda-4041ef5bc080","Type":"ContainerDied","Data":"24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631"} Nov 02 20:43:44.591253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.591198 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" event={"ID":"fc9eb207-3df4-4ba5-acda-4041ef5bc080","Type":"ContainerDied","Data":"9e8a6b193b922793d16125f9374cb4c47062d575231646b00b38da2d66a84145"} Nov 02 20:43:44.591309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.591274 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs" Nov 02 20:43:44.594012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.593604 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" Nov 02 20:43:44.594012 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.593605 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj" event={"ID":"54d935d8-83fb-458c-a4ef-f0a0baf669be","Type":"ContainerDied","Data":"2be8ac2ceae3cfc325025372cd53c68a784c717a3806d307b2f3ce53f8acc6e4"} Nov 02 20:43:44.600641 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.600618 5420 scope.go:122] "RemoveContainer" containerID="70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2" Nov 02 20:43:44.601004 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:44.600979 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2\": container with ID starting with 70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2 not found: ID does not exist" containerID="70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2" Nov 02 20:43:44.601108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.601012 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2"} err="failed to get container status \"70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2\": rpc error: code = NotFound desc = could not find container \"70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2\": container with ID starting with 70ea812c1b501f41f475a3ddf33c18682c5c8b35810292d8f7723b729f30ced2 not found: ID does not exist" Nov 02 20:43:44.601108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.601037 5420 scope.go:122] "RemoveContainer" containerID="3b8a0ec1db1b1e898f2a5a4227cc951786915e7887ba5d2fc547f3b709e1a3ce" Nov 02 20:43:44.612246 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.612225 5420 scope.go:122] "RemoveContainer" containerID="24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631" Nov 02 20:43:44.621366 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.621346 5420 scope.go:122] "RemoveContainer" containerID="d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1" Nov 02 20:43:44.629918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.629899 5420 scope.go:122] "RemoveContainer" containerID="24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631" Nov 02 20:43:44.630180 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:44.630161 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631\": container with ID starting with 24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631 not found: ID does not exist" containerID="24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631" Nov 02 20:43:44.630248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.630187 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631"} err="failed to get container status \"24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631\": rpc error: code = NotFound desc = could not find container \"24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631\": container with ID starting with 24c7fa7bc667e2f6f04ba5cc10723f6ad7141050b60fa6a2ec18f954f86a4631 not found: ID does not exist" Nov 02 20:43:44.630248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.630203 5420 scope.go:122] "RemoveContainer" containerID="d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1" Nov 02 20:43:44.630470 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:44.630447 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1\": container with ID starting with d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1 not found: ID does not exist" containerID="d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1" Nov 02 20:43:44.630516 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.630481 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1"} err="failed to get container status \"d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1\": rpc error: code = NotFound desc = could not find container \"d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1\": container with ID starting with d5925197a433e7e92ad465ac7117eb1f584e98ec8a2ce515c963d0e583e4c6e1 not found: ID does not exist" Nov 02 20:43:44.630516 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.630502 5420 scope.go:122] "RemoveContainer" containerID="3c9af6f70ec9bef9bd8b528a82938230158e15b0eb8ee4f508356be71b01f9f1" Nov 02 20:43:44.744740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.744656 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs"] Nov 02 20:43:44.792561 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.792537 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-mn9fs"] Nov 02 20:43:44.851103 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.851073 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth"] Nov 02 20:43:44.861611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.861571 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-zktth"] Nov 02 20:43:44.960023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.959990 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx"] Nov 02 20:43:44.964961 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:44.964934 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-hc5rx"] Nov 02 20:43:45.072502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:45.072477 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj"] Nov 02 20:43:45.078004 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:45.077980 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-bce768-openshift-default-86f65b9c9f-jqbhj"] Nov 02 20:43:45.775497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:45.775466 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f68223-bd3d-496b-a498-be15cc8bdec7" path="/var/lib/kubelet/pods/52f68223-bd3d-496b-a498-be15cc8bdec7/volumes" Nov 02 20:43:45.775959 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:45.775944 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" path="/var/lib/kubelet/pods/54d935d8-83fb-458c-a4ef-f0a0baf669be/volumes" Nov 02 20:43:45.776368 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:45.776355 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636700c4-2720-4f52-8043-d15cead9abf4" path="/var/lib/kubelet/pods/636700c4-2720-4f52-8043-d15cead9abf4/volumes" Nov 02 20:43:45.777075 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:45.777054 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" path="/var/lib/kubelet/pods/cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1/volumes" Nov 02 20:43:45.777508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:45.777494 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" path="/var/lib/kubelet/pods/fc9eb207-3df4-4ba5-acda-4041ef5bc080/volumes" Nov 02 20:43:47.516859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.516832 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:43:47.622493 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.621963 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-kube-api-access-lj698\" (UniqueName: \"kubernetes.io/projected/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-kube-api-access-lj698\") pod \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\" (UID: \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\") " Nov 02 20:43:47.622493 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.622011 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-plugins-config-volume\") pod \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\" (UID: \"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b\") " Nov 02 20:43:47.622996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.622892 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-plugins-config-volume" pod "dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" (UID: "dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:47.636695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.626316 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-kube-api-access-lj698" pod "dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" (UID: "dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b"). InnerVolumeSpecName "kube-api-access-lj698". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:47.636695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.632068 5420 generic.go:356] "Generic (PLEG): container finished" podID="dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" containerID="26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c" exitCode=0 Nov 02 20:43:47.636695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.632264 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" event={"ID":"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b","Type":"ContainerDied","Data":"26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c"} Nov 02 20:43:47.636695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.632342 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" event={"ID":"dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b","Type":"ContainerDied","Data":"bd9e75efe8a8a994c047da766bffbf29a5efcb76616c3f90f4f6526f0b35a665"} Nov 02 20:43:47.636695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.632368 5420 scope.go:122] "RemoveContainer" containerID="26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c" Nov 02 20:43:47.636695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.632528 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2" Nov 02 20:43:47.653309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.653220 5420 scope.go:122] "RemoveContainer" containerID="26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c" Nov 02 20:43:47.653614 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:47.653581 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c\": container with ID starting with 26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c not found: ID does not exist" containerID="26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c" Nov 02 20:43:47.653715 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.653627 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c"} err="failed to get container status \"26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c\": rpc error: code = NotFound desc = could not find container \"26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c\": container with ID starting with 26d083cdcfebbdcd1aad3a8dd2ea14dc5dfc139c979e9b45feb27d40f976cc4c not found: ID does not exist" Nov 02 20:43:47.724993 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.724961 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj698\" (UniqueName: \"kubernetes.io/projected/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-kube-api-access-lj698\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:47.724993 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.724993 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:47.805833 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.805804 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2"] Nov 02 20:43:47.807536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:47.807502 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-694cee-6f5d875594-r5qz2"] Nov 02 20:43:49.382470 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:49.382438 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2"] Nov 02 20:43:49.382931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:49.382646 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" containerID="cri-o://cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e" gracePeriod=30 Nov 02 20:43:49.776333 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:49.776256 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" path="/var/lib/kubelet/pods/dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b/volumes" Nov 02 20:43:53.553448 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:53.553411 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn"] Nov 02 20:43:53.553935 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:53.553694 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" containerID="cri-o://e7fbd18432301be0b036cf894c072148f9e85093f33ab36437f98417f68ad712" gracePeriod=30 Nov 02 20:43:53.599932 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:53.599899 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t"] Nov 02 20:43:53.600149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:53.600109 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" containerID="cri-o://555e4899e19d1415c2882f190398d485410dcfddbc6203072fec4fec657bed7b" gracePeriod=30 Nov 02 20:43:53.603456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:53.603431 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h"] Nov 02 20:43:53.603652 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:53.603619 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" containerID="cri-o://0b40e361510a7ce3a1137675341cc18b6149d86c837c8d3045c265cf552ce1a9" gracePeriod=30 Nov 02 20:43:53.645376 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:53.645350 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b"] Nov 02 20:43:53.645574 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:53.645552 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" containerID="cri-o://a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874" gracePeriod=30 Nov 02 20:43:54.427383 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.427355 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 20:43:54.427478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.427418 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 20:43:54.652830 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.652772 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:43:54.705637 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.705609 5420 generic.go:356] "Generic (PLEG): container finished" podID="5490c04a-3123-4317-8906-a95d09bd1c13" containerID="cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e" exitCode=0 Nov 02 20:43:54.705803 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.705697 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" Nov 02 20:43:54.705803 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.705720 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" event={"ID":"5490c04a-3123-4317-8906-a95d09bd1c13","Type":"ContainerDied","Data":"cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e"} Nov 02 20:43:54.705803 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.705786 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2" event={"ID":"5490c04a-3123-4317-8906-a95d09bd1c13","Type":"ContainerDied","Data":"59383a399d7359c1ac32eb9a228f3bb71839b8f6d7190112f34e876618b632d2"} Nov 02 20:43:54.705973 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.705814 5420 scope.go:122] "RemoveContainer" containerID="cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e" Nov 02 20:43:54.716143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.716127 5420 scope.go:122] "RemoveContainer" containerID="cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e" Nov 02 20:43:54.716401 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:54.716374 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e\": container with ID starting with cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e not found: ID does not exist" containerID="cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e" Nov 02 20:43:54.716469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.716412 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e"} err="failed to get container status \"cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e\": rpc error: code = NotFound desc = could not find container \"cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e\": container with ID starting with cadc5f8ccae026b14c7503763eca9ff1dc8890ffe6d56cac89092a9d7bc80d8e not found: ID does not exist" Nov 02 20:43:54.743830 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.743806 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-credential-socket\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.743938 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.743894 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5490c04a-3123-4317-8906-a95d09bd1c13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5490c04a-3123-4317-8906-a95d09bd1c13-istiod-ca-cert\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.744002 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.743993 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-certs\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.744062 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744032 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-data\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.744120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744061 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-envoy\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.744120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744081 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-credential-socket" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.744120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744094 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-istio-token\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-istio-token\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.744258 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744240 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-socket\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.744309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744266 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-data" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.744309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744279 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-certs" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.744309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744298 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5490c04a-3123-4317-8906-a95d09bd1c13-istiod-ca-cert" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:54.744441 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744325 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-kube-api-access-j8xxt\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-kube-api-access-j8xxt\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.744441 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744367 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5490c04a-3123-4317-8906-a95d09bd1c13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5490c04a-3123-4317-8906-a95d09bd1c13-istio-podinfo\") pod \"5490c04a-3123-4317-8906-a95d09bd1c13\" (UID: \"5490c04a-3123-4317-8906-a95d09bd1c13\") " Nov 02 20:43:54.744559 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744536 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-socket" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.744970 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744947 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.744970 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744972 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5490c04a-3123-4317-8906-a95d09bd1c13-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.745151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744981 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.745151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744988 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.745151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.744996 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.746381 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.746361 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5490c04a-3123-4317-8906-a95d09bd1c13-istio-podinfo" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:54.746463 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.746441 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-kube-api-access-j8xxt" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "kube-api-access-j8xxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:54.746603 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.746584 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-envoy" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:54.746603 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.746587 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-istio-token" pod "5490c04a-3123-4317-8906-a95d09bd1c13" (UID: "5490c04a-3123-4317-8906-a95d09bd1c13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:54.846234 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.846199 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.846234 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.846232 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8xxt\" (UniqueName: \"kubernetes.io/projected/5490c04a-3123-4317-8906-a95d09bd1c13-kube-api-access-j8xxt\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.846509 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.846249 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5490c04a-3123-4317-8906-a95d09bd1c13-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:54.846509 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:54.846262 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5490c04a-3123-4317-8906-a95d09bd1c13-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:55.127918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:55.127884 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2"] Nov 02 20:43:55.133113 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:55.133079 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-qh5z2"] Nov 02 20:43:55.777223 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:55.777189 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" path="/var/lib/kubelet/pods/5490c04a-3123-4317-8906-a95d09bd1c13/volumes" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.881910 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4"] Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.882875 5420 cpu_manager.go:414] "containerMap: removing container" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.882896 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.882960 5420 cpu_manager.go:414] "containerMap: removing container" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.882968 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883021 5420 cpu_manager.go:414] "containerMap: removing container" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883029 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883040 5420 cpu_manager.go:414] "containerMap: removing container" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883056 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883073 5420 cpu_manager.go:414] "containerMap: removing container" podUID="52f68223-bd3d-496b-a498-be15cc8bdec7" containerName="payload-pre-processing" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883081 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f68223-bd3d-496b-a498-be15cc8bdec7" containerName="payload-pre-processing" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883095 5420 cpu_manager.go:414] "containerMap: removing container" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883106 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883130 5420 cpu_manager.go:414] "containerMap: removing container" podUID="dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" containerName="payload-processing" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883137 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" containerName="payload-processing" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883169 5420 cpu_manager.go:414] "containerMap: removing container" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883176 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883197 5420 cpu_manager.go:414] "containerMap: removing container" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883205 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883372 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="636700c4-2720-4f52-8043-d15cead9abf4" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883386 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="5490c04a-3123-4317-8906-a95d09bd1c13" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883398 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e1d983-6558-405b-ad3a-58f3121815e6" containerName="main" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883406 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc79e0e3-64d1-4bbd-9bf8-dfb9ffe81cc1" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883416 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f68223-bd3d-496b-a498-be15cc8bdec7" containerName="payload-pre-processing" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883425 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883433 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2ce8ee-1763-4542-8c70-7cf8a2a4cc6b" containerName="payload-processing" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883448 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc9eb207-3df4-4ba5-acda-4041ef5bc080" containerName="istio-proxy" Nov 02 20:43:56.885722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.883456 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d935d8-83fb-458c-a4ef-f0a0baf669be" containerName="istio-proxy" Nov 02 20:43:56.888305 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.888283 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:56.969695 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.959977 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4"] Nov 02 20:43:56.987913 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.987885 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-039b89\"" Nov 02 20:43:56.988291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.988273 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-039b89-dockercfg-wrkmf\"" Nov 02 20:43:56.996630 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.996590 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4\" (UID: \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:56.997037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:56.997016 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2pv7\" (UniqueName: \"kubernetes.io/projected/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-kube-api-access-s2pv7\") pod \"payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4\" (UID: \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:57.101611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.101573 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4\" (UID: \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:57.101800 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.101779 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2pv7\" (UniqueName: \"kubernetes.io/projected/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-kube-api-access-s2pv7\") pod \"payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4\" (UID: \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:57.103205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.103177 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4\" (UID: \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:57.172020 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.171945 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2pv7\" (UniqueName: \"kubernetes.io/projected/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-kube-api-access-s2pv7\") pod \"payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4\" (UID: \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:57.220307 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.220281 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:57.387776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.387331 5420 patch_prober.go:29] interesting pod/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:43:57.387776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.387386 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:43:57.602076 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:43:57.602043 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4761389_0a5c_4fbf_a988_3a5c7e45c80d.slice/crio-8f39652f9a540c8dfbe5c69e1eabccaaf2de46eb5f5979a8f291c274ef37f599 WatchSource:0}: Error finding container 8f39652f9a540c8dfbe5c69e1eabccaaf2de46eb5f5979a8f291c274ef37f599: Status 404 returned error can't find the container with id 8f39652f9a540c8dfbe5c69e1eabccaaf2de46eb5f5979a8f291c274ef37f599 Nov 02 20:43:57.602261 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.602107 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4"] Nov 02 20:43:57.731173 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.731144 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" event={"ID":"f4761389-0a5c-4fbf-a988-3a5c7e45c80d","Type":"ContainerStarted","Data":"c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94"} Nov 02 20:43:57.731311 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.731195 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:43:57.731311 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.731219 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" event={"ID":"f4761389-0a5c-4fbf-a988-3a5c7e45c80d","Type":"ContainerStarted","Data":"8f39652f9a540c8dfbe5c69e1eabccaaf2de46eb5f5979a8f291c274ef37f599"} Nov 02 20:43:57.841376 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:57.841327 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" podStartSLOduration=1.8413130020000001 podStartE2EDuration="1.841313002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:43:57.837864414 +0000 UTC m=+1710.202475618" watchObservedRunningTime="2026-11-02 20:43:57.841313002 +0000 UTC m=+1710.205924206" Nov 02 20:43:58.741578 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:58.741550 5420 generic.go:356] "Generic (PLEG): container finished" podID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerID="e7fbd18432301be0b036cf894c072148f9e85093f33ab36437f98417f68ad712" exitCode=0 Nov 02 20:43:58.744040 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:58.741623 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" event={"ID":"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b","Type":"ContainerDied","Data":"e7fbd18432301be0b036cf894c072148f9e85093f33ab36437f98417f68ad712"} Nov 02 20:43:58.744459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:58.744442 5420 generic.go:356] "Generic (PLEG): container finished" podID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerID="0b40e361510a7ce3a1137675341cc18b6149d86c837c8d3045c265cf552ce1a9" exitCode=0 Nov 02 20:43:58.744544 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:58.744530 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" event={"ID":"d3ffb888-80b2-42d3-9711-d8b09ffc5796","Type":"ContainerDied","Data":"0b40e361510a7ce3a1137675341cc18b6149d86c837c8d3045c265cf552ce1a9"} Nov 02 20:43:58.746881 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:58.746847 5420 generic.go:356] "Generic (PLEG): container finished" podID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerID="555e4899e19d1415c2882f190398d485410dcfddbc6203072fec4fec657bed7b" exitCode=0 Nov 02 20:43:58.747169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:58.747144 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" event={"ID":"8d0d5bf8-5843-4ba4-9652-e29a1f05c965","Type":"ContainerDied","Data":"555e4899e19d1415c2882f190398d485410dcfddbc6203072fec4fec657bed7b"} Nov 02 20:43:58.957523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:58.957501 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:43:59.008713 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.008682 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:43:59.079120 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.079093 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:43:59.083279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.083255 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:43:59.131852 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.131768 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-certs\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.132033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.131852 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-data\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.132033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.131949 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-certs\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.132033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.131996 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-credential-socket\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.132205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132033 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-envoy\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.132205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132080 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-socket\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.132205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132119 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-token\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-token\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.132205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132139 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-certs" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.132205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132194 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-token\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-token\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.132447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132230 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-credential-socket\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.132447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132306 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-kube-api-access-4c8mq\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-kube-api-access-4c8mq\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.132447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132349 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-podinfo\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.132447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132374 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-credential-socket" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.132447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132395 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istiod-ca-cert\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.132447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132445 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-kube-api-access-wvvpt\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-kube-api-access-wvvpt\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132517 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-data\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132577 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-podinfo\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132579 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-data" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132605 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-envoy\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132640 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-socket\") pod \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\" (UID: \"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b\") " Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132697 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istiod-ca-cert\") pod \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\" (UID: \"d3ffb888-80b2-42d3-9711-d8b09ffc5796\") " Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.132782 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-certs" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.133601 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.133626 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.133640 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.133655 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.134042 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istiod-ca-cert" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.134383 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istiod-ca-cert" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.135086 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-envoy" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.135556 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-podinfo" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.135819 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-credential-socket" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.136880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.136851 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-kube-api-access-wvvpt" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "kube-api-access-wvvpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:59.139787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.137126 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-data" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.139787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.137151 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-socket" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.139787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.137210 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-socket" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.139787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.138048 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-kube-api-access-4c8mq" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "kube-api-access-4c8mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:59.139787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.138213 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-token" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:59.139787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.139326 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-token" pod "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" (UID: "15fd9aed-bf60-4b23-a5f1-c166b2c48f2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:59.139787 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.139594 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-podinfo" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:59.140971 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.140947 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-envoy" pod "d3ffb888-80b2-42d3-9711-d8b09ffc5796" (UID: "d3ffb888-80b2-42d3-9711-d8b09ffc5796"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.239273 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.239235 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-credential-socket\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.239854 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.239808 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-envoy\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.240145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240118 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-kube-api-access-d2cmg\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-kube-api-access-d2cmg\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.240254 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240232 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-socket\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.240330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240282 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-data\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.240412 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240326 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-envoy\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.240412 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240367 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-data\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.240567 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240445 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-certs\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.240567 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240492 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-credential-socket\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.240567 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240529 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-kube-api-access-twncv\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-kube-api-access-twncv\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.240773 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240623 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istiod-ca-cert\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.240773 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240714 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-token\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-token\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.240898 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240814 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fea91a2-c7af-4431-aa75-cf955eec57c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fea91a2-c7af-4431-aa75-cf955eec57c6-istiod-ca-cert\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.240898 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240851 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-token\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-token\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.240898 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240887 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-certs\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.241117 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240921 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-podinfo\") pod \"5fea91a2-c7af-4431-aa75-cf955eec57c6\" (UID: \"5fea91a2-c7af-4431-aa75-cf955eec57c6\") " Nov 02 20:43:59.241117 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.240979 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-socket\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.241117 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.241024 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-podinfo\") pod \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\" (UID: \"8d0d5bf8-5843-4ba4-9652-e29a1f05c965\") " Nov 02 20:43:59.241272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.241137 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-credential-socket" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242185 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242214 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242229 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242241 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242261 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242273 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8mq\" (UniqueName: \"kubernetes.io/projected/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-kube-api-access-4c8mq\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242287 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242299 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242316 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvvpt\" (UniqueName: \"kubernetes.io/projected/d3ffb888-80b2-42d3-9711-d8b09ffc5796-kube-api-access-wvvpt\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242328 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242341 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242353 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242370 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242382 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3ffb888-80b2-42d3-9711-d8b09ffc5796-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.242690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.242396 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.243484 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.243108 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-socket" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.243631 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.243605 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-data" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.243869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.243844 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-certs" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.244055 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.244028 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-credential-socket" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.244146 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.244035 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-certs" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.244146 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.244122 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istiod-ca-cert" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:59.244248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.244177 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-kube-api-access-twncv" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "kube-api-access-twncv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:59.244248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.244207 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-data" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.244405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.244374 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fea91a2-c7af-4431-aa75-cf955eec57c6-istiod-ca-cert" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:43:59.244649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.244626 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-socket" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.246212 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.246171 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-podinfo" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:59.249037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.249015 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-token" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:59.255593 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.255557 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-envoy" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.255795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.255776 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-envoy" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:43:59.260343 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.259617 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-podinfo" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:43:59.268587 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.268545 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-kube-api-access-d2cmg" pod "5fea91a2-c7af-4431-aa75-cf955eec57c6" (UID: "5fea91a2-c7af-4431-aa75-cf955eec57c6"). InnerVolumeSpecName "kube-api-access-d2cmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:59.268704 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.268593 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-token" pod "8d0d5bf8-5843-4ba4-9652-e29a1f05c965" (UID: "8d0d5bf8-5843-4ba4-9652-e29a1f05c965"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345689 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345724 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2cmg\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-kube-api-access-d2cmg\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345740 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345753 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345767 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345778 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345792 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fea91a2-c7af-4431-aa75-cf955eec57c6-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345805 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345819 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twncv\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-kube-api-access-twncv\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345832 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345844 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345857 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fea91a2-c7af-4431-aa75-cf955eec57c6-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345870 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345884 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345896 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fea91a2-c7af-4431-aa75-cf955eec57c6-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345952 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.346693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.345964 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0d5bf8-5843-4ba4-9652-e29a1f05c965-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:43:59.756350 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.756324 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" Nov 02 20:43:59.756816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.756349 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t" event={"ID":"8d0d5bf8-5843-4ba4-9652-e29a1f05c965","Type":"ContainerDied","Data":"4eed4d796448413aa933c6930174a8c903cf8f41535efbc129a73eedd1f68752"} Nov 02 20:43:59.756816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.756448 5420 scope.go:122] "RemoveContainer" containerID="555e4899e19d1415c2882f190398d485410dcfddbc6203072fec4fec657bed7b" Nov 02 20:43:59.758023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.757948 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" Nov 02 20:43:59.758023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.757972 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn" event={"ID":"15fd9aed-bf60-4b23-a5f1-c166b2c48f2b","Type":"ContainerDied","Data":"fd680d7c1a2c185c7b32d4caeedd0e1df1bea59eae843adde4e4c95027e4e5ff"} Nov 02 20:43:59.759721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.759474 5420 generic.go:356] "Generic (PLEG): container finished" podID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerID="a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874" exitCode=0 Nov 02 20:43:59.759721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.759568 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" Nov 02 20:43:59.759721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.759619 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" event={"ID":"5fea91a2-c7af-4431-aa75-cf955eec57c6","Type":"ContainerDied","Data":"a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874"} Nov 02 20:43:59.759721 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.759653 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b" event={"ID":"5fea91a2-c7af-4431-aa75-cf955eec57c6","Type":"ContainerDied","Data":"d91b056b58111507aa3a10d34f71d82ece97836e647f23505ed38be85a8a852e"} Nov 02 20:43:59.765736 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.761416 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" event={"ID":"d3ffb888-80b2-42d3-9711-d8b09ffc5796","Type":"ContainerDied","Data":"0bce8d37d3b9b458b32cb5c5a9d2315da495f29f2212eaa1f668e61ead5f992d"} Nov 02 20:43:59.765736 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.761736 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h" Nov 02 20:43:59.777889 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.777868 5420 scope.go:122] "RemoveContainer" containerID="e7fbd18432301be0b036cf894c072148f9e85093f33ab36437f98417f68ad712" Nov 02 20:43:59.791953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.791932 5420 scope.go:122] "RemoveContainer" containerID="a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874" Nov 02 20:43:59.803270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.803243 5420 scope.go:122] "RemoveContainer" containerID="a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874" Nov 02 20:43:59.803567 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:43:59.803544 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874\": container with ID starting with a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874 not found: ID does not exist" containerID="a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874" Nov 02 20:43:59.803630 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.803580 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874"} err="failed to get container status \"a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874\": rpc error: code = NotFound desc = could not find container \"a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874\": container with ID starting with a7630aea54a2cb80b971a8aa35baac808705762a030f26310bbd688ac4877874 not found: ID does not exist" Nov 02 20:43:59.803630 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.803601 5420 scope.go:122] "RemoveContainer" containerID="0b40e361510a7ce3a1137675341cc18b6149d86c837c8d3045c265cf552ce1a9" Nov 02 20:43:59.914250 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.914194 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h"] Nov 02 20:43:59.959970 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:43:59.959946 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-glm8h"] Nov 02 20:44:00.024433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:00.024402 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t"] Nov 02 20:44:00.027785 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:00.027747 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-kk99t"] Nov 02 20:44:00.129996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:00.129959 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn"] Nov 02 20:44:00.136220 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:00.136195 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-cfmtn"] Nov 02 20:44:00.236846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:00.236782 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b"] Nov 02 20:44:00.242519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:00.242496 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-694cee-openshift-default-84bcc8dfc7-k9f6b"] Nov 02 20:44:01.792739 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:01.792700 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" path="/var/lib/kubelet/pods/15fd9aed-bf60-4b23-a5f1-c166b2c48f2b/volumes" Nov 02 20:44:01.793433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:01.793413 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" path="/var/lib/kubelet/pods/5fea91a2-c7af-4431-aa75-cf955eec57c6/volumes" Nov 02 20:44:01.794023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:01.793968 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" path="/var/lib/kubelet/pods/8d0d5bf8-5843-4ba4-9652-e29a1f05c965/volumes" Nov 02 20:44:01.794900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:01.794881 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" path="/var/lib/kubelet/pods/d3ffb888-80b2-42d3-9711-d8b09ffc5796/volumes" Nov 02 20:44:02.413697 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.413654 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr"] Nov 02 20:44:02.414372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414349 5420 cpu_manager.go:414] "containerMap: removing container" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" Nov 02 20:44:02.414372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414369 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" Nov 02 20:44:02.414541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414393 5420 cpu_manager.go:414] "containerMap: removing container" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" Nov 02 20:44:02.414541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414400 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" Nov 02 20:44:02.414541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414431 5420 cpu_manager.go:414] "containerMap: removing container" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" Nov 02 20:44:02.414541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414439 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" Nov 02 20:44:02.414541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414455 5420 cpu_manager.go:414] "containerMap: removing container" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" Nov 02 20:44:02.414541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414462 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" Nov 02 20:44:02.414741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414634 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fea91a2-c7af-4431-aa75-cf955eec57c6" containerName="istio-proxy" Nov 02 20:44:02.414741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414649 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="15fd9aed-bf60-4b23-a5f1-c166b2c48f2b" containerName="istio-proxy" Nov 02 20:44:02.414741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414659 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0d5bf8-5843-4ba4-9652-e29a1f05c965" containerName="istio-proxy" Nov 02 20:44:02.414741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.414666 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ffb888-80b2-42d3-9711-d8b09ffc5796" containerName="istio-proxy" Nov 02 20:44:02.418776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.418756 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.471429 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.471406 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-4166b0-serving-cert\"" Nov 02 20:44:02.495793 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.495770 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29tt9\" (UniqueName: \"kubernetes.io/projected/9e453b0a-c58d-4733-93ec-935015f97d41-kube-api-access-29tt9\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.495907 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.495824 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.496102 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.496077 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e453b0a-c58d-4733-93ec-935015f97d41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.496236 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.496212 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.521631 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.521600 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr"] Nov 02 20:44:02.597665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.597636 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e453b0a-c58d-4733-93ec-935015f97d41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.597821 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.597725 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.597821 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.597767 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29tt9\" (UniqueName: \"kubernetes.io/projected/9e453b0a-c58d-4733-93ec-935015f97d41-kube-api-access-29tt9\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.597821 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.597797 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.597973 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:02.597823 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-serving-cert: secret "maas-api-e2e-shared-a-w3-4166b0-serving-cert" not found Nov 02 20:44:02.597973 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:02.597903 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls podName:9e453b0a-c58d-4733-93ec-935015f97d41 nodeName:}" failed. No retries permitted until 2026-11-02 20:44:03.097882256 +0000 UTC m=+1715.462493453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls") pod "maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" (UID: "9e453b0a-c58d-4733-93ec-935015f97d41") : secret "maas-api-e2e-shared-a-w3-4166b0-serving-cert" not found Nov 02 20:44:02.598331 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.598312 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e453b0a-c58d-4733-93ec-935015f97d41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.600131 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.600111 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:02.679737 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:02.679426 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:44:02.700978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:02.700342 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29tt9\" (UniqueName: \"kubernetes.io/projected/9e453b0a-c58d-4733-93ec-935015f97d41-kube-api-access-29tt9\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:03.107074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:03.107040 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:03.109546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:03.109519 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:03.331304 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:03.331275 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:03.709481 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:03.709437 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr"] Nov 02 20:44:03.711494 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:44:03.711468 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e453b0a_c58d_4733_93ec_935015f97d41.slice/crio-041c3a6c47c05171bd6ba7027d0b83ba355302606dd9207cb8c09860b9865c08 WatchSource:0}: Error finding container 041c3a6c47c05171bd6ba7027d0b83ba355302606dd9207cb8c09860b9865c08: Status 404 returned error can't find the container with id 041c3a6c47c05171bd6ba7027d0b83ba355302606dd9207cb8c09860b9865c08 Nov 02 20:44:03.713251 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:03.713235 5420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 20:44:03.816040 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:03.816012 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" event={"ID":"9e453b0a-c58d-4733-93ec-935015f97d41","Type":"ContainerStarted","Data":"041c3a6c47c05171bd6ba7027d0b83ba355302606dd9207cb8c09860b9865c08"} Nov 02 20:44:04.824335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:04.824290 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" event={"ID":"9e453b0a-c58d-4733-93ec-935015f97d41","Type":"ContainerStarted","Data":"7198163d617508ede8d4354783dc9cc2a16a0aa715b2194627a2aa077b61f742"} Nov 02 20:44:04.824800 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:04.824454 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:04.942890 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:04.942800 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" podStartSLOduration=2.007554213 podStartE2EDuration="2.942787798s" totalImagesPullingTime="935.233585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:04.940697436 +0000 UTC m=+1717.305308635" watchObservedRunningTime="2026-11-02 20:44:04.942787798 +0000 UTC m=+1717.307399003" Nov 02 20:44:08.660788 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:08.660709 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn"] Nov 02 20:44:08.661310 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:08.660930 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" containerID="cri-o://ddd4bcdc8f5cc7d0a3c94a65f16316f8124cb34cd2368924ba927b7e8e36ab1f" gracePeriod=30 Nov 02 20:44:08.751442 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:08.751412 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:44:10.837429 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:10.837398 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:44:13.891901 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:13.891865 5420 generic.go:356] "Generic (PLEG): container finished" podID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerID="ddd4bcdc8f5cc7d0a3c94a65f16316f8124cb34cd2368924ba927b7e8e36ab1f" exitCode=0 Nov 02 20:44:13.892341 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:13.891907 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" event={"ID":"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553","Type":"ContainerDied","Data":"ddd4bcdc8f5cc7d0a3c94a65f16316f8124cb34cd2368924ba927b7e8e36ab1f"} Nov 02 20:44:13.941584 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:13.941530 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:44:14.061963 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.061935 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-token\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-token\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062136 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.061982 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-socket\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062136 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062058 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-envoy\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062259 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062196 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-podinfo\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062307 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062268 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-kube-api-access-wtx9v\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-kube-api-access-wtx9v\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062322 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istiod-ca-cert\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062265 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-socket" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.062458 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062402 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-data\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062508 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062497 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-credential-socket\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062560 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062546 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-certs\") pod \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\" (UID: \"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553\") " Nov 02 20:44:14.062766 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062740 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-data" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.062868 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062836 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istiod-ca-cert" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:44:14.062989 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062950 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-certs" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.063051 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.062983 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-credential-socket" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.063256 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.063202 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.063256 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.063225 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.063256 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.063235 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.063256 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.063244 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.063256 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.063256 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.064268 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.064244 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-token" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:14.064584 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.064564 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-podinfo" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:44:14.064969 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.064948 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-kube-api-access-wtx9v" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "kube-api-access-wtx9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:14.065050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.064974 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-envoy" pod "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" (UID: "7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:14.164107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.164076 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtx9v\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-kube-api-access-wtx9v\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.164107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.164105 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.164107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.164114 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.164320 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.164122 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:14.901702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.901639 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" event={"ID":"7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553","Type":"ContainerDied","Data":"2d111ed1716c067c920be4397710ba2a591b62bba4ac82cd16bfb29afc320903"} Nov 02 20:44:14.902154 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.901715 5420 scope.go:122] "RemoveContainer" containerID="ddd4bcdc8f5cc7d0a3c94a65f16316f8124cb34cd2368924ba927b7e8e36ab1f" Nov 02 20:44:14.902154 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:14.901634 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn" Nov 02 20:44:15.103836 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:15.103806 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn"] Nov 02 20:44:15.149266 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:15.149239 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-825pn"] Nov 02 20:44:15.775988 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:15.775958 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" path="/var/lib/kubelet/pods/7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553/volumes" Nov 02 20:44:19.205065 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.205033 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv"] Nov 02 20:44:19.205802 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.205783 5420 cpu_manager.go:414] "containerMap: removing container" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" Nov 02 20:44:19.205802 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.205802 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" Nov 02 20:44:19.205945 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.205935 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ffa4cfa-7f88-4444-8a8b-1ac3c26f7553" containerName="istio-proxy" Nov 02 20:44:19.208608 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.208593 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.259194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.259122 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-1620dc-serving-cert\"" Nov 02 20:44:19.309122 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.309088 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv"] Nov 02 20:44:19.328996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.328966 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.329143 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.329011 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6zr5\" (UniqueName: \"kubernetes.io/projected/90ab3a61-5a80-4946-841b-c937011affc4-kube-api-access-v6zr5\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.329237 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.329216 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ab3a61-5a80-4946-841b-c937011affc4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.329355 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.329327 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.430529 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.430497 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ab3a61-5a80-4946-841b-c937011affc4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.430719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.430567 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.430719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.430592 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.430719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.430620 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zr5\" (UniqueName: \"kubernetes.io/projected/90ab3a61-5a80-4946-841b-c937011affc4-kube-api-access-v6zr5\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.430908 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:19.430743 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-serving-cert: secret "maas-api-e2e-shared-a-w6-1620dc-serving-cert" not found Nov 02 20:44:19.430908 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:19.430819 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls podName:90ab3a61-5a80-4946-841b-c937011affc4 nodeName:}" failed. No retries permitted until 2026-11-02 20:44:19.930798629 +0000 UTC m=+1732.295409817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls") pod "maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" (UID: "90ab3a61-5a80-4946-841b-c937011affc4") : secret "maas-api-e2e-shared-a-w6-1620dc-serving-cert" not found Nov 02 20:44:19.431266 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.431235 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ab3a61-5a80-4946-841b-c937011affc4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.433023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.432999 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.487809 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.487729 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6zr5\" (UniqueName: \"kubernetes.io/projected/90ab3a61-5a80-4946-841b-c937011affc4-kube-api-access-v6zr5\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.936400 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.936368 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:19.938864 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:19.938841 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:20.121834 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:20.121807 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:20.494219 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:20.494189 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv"] Nov 02 20:44:20.497795 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:44:20.497764 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ab3a61_5a80_4946_841b_c937011affc4.slice/crio-d96d458ea2e1ecfbefdea40ebac383a39bef473904d1aa42abd523086ba53a3e WatchSource:0}: Error finding container d96d458ea2e1ecfbefdea40ebac383a39bef473904d1aa42abd523086ba53a3e: Status 404 returned error can't find the container with id d96d458ea2e1ecfbefdea40ebac383a39bef473904d1aa42abd523086ba53a3e Nov 02 20:44:20.947450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:20.947419 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" event={"ID":"90ab3a61-5a80-4946-841b-c937011affc4","Type":"ContainerStarted","Data":"d96d458ea2e1ecfbefdea40ebac383a39bef473904d1aa42abd523086ba53a3e"} Nov 02 20:44:21.955937 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:21.955907 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" event={"ID":"90ab3a61-5a80-4946-841b-c937011affc4","Type":"ContainerStarted","Data":"f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb"} Nov 02 20:44:21.956450 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:21.956027 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:22.078936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:22.078888 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" podStartSLOduration=2.235578071 podStartE2EDuration="3.078875581s" totalImagesPullingTime="843.29751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:22.075945244 +0000 UTC m=+1734.440556448" watchObservedRunningTime="2026-11-02 20:44:22.078875581 +0000 UTC m=+1734.443486785" Nov 02 20:44:27.968725 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:27.968687 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:44:32.658561 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:32.658525 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:44:33.619519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.619476 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc"] Nov 02 20:44:33.625387 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.625365 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:33.675666 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.674960 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-18fd4c\"" Nov 02 20:44:33.675666 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.675355 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-18fd4c-dockercfg-szxpn\"" Nov 02 20:44:33.682200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.682170 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc"] Nov 02 20:44:33.720706 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.719928 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef16c845-34db-4717-bae8-173ec711aa55-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc\" (UID: \"ef16c845-34db-4717-bae8-173ec711aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:33.720706 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.720129 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrf8f\" (UniqueName: \"kubernetes.io/projected/ef16c845-34db-4717-bae8-173ec711aa55-kube-api-access-rrf8f\") pod \"payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc\" (UID: \"ef16c845-34db-4717-bae8-173ec711aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:33.839602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.839558 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef16c845-34db-4717-bae8-173ec711aa55-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc\" (UID: \"ef16c845-34db-4717-bae8-173ec711aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:33.840195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.840167 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrf8f\" (UniqueName: \"kubernetes.io/projected/ef16c845-34db-4717-bae8-173ec711aa55-kube-api-access-rrf8f\") pod \"payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc\" (UID: \"ef16c845-34db-4717-bae8-173ec711aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:33.841790 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.841765 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef16c845-34db-4717-bae8-173ec711aa55-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc\" (UID: \"ef16c845-34db-4717-bae8-173ec711aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:33.860435 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.858480 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw"] Nov 02 20:44:33.861797 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.861780 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw"] Nov 02 20:44:33.861976 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.861966 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:33.897512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.897485 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrf8f\" (UniqueName: \"kubernetes.io/projected/ef16c845-34db-4717-bae8-173ec711aa55-kube-api-access-rrf8f\") pod \"payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc\" (UID: \"ef16c845-34db-4717-bae8-173ec711aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:33.939969 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.939945 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:33.941531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.941506 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf67e9e-2c20-42bd-869d-e4637795eb21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw\" (UID: \"3bf67e9e-2c20-42bd-869d-e4637795eb21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:33.941765 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:33.941749 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q54hm\" (UniqueName: \"kubernetes.io/projected/3bf67e9e-2c20-42bd-869d-e4637795eb21-kube-api-access-q54hm\") pod \"payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw\" (UID: \"3bf67e9e-2c20-42bd-869d-e4637795eb21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:34.042655 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:34.042620 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf67e9e-2c20-42bd-869d-e4637795eb21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw\" (UID: \"3bf67e9e-2c20-42bd-869d-e4637795eb21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:34.042819 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:34.042770 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q54hm\" (UniqueName: \"kubernetes.io/projected/3bf67e9e-2c20-42bd-869d-e4637795eb21-kube-api-access-q54hm\") pod \"payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw\" (UID: \"3bf67e9e-2c20-42bd-869d-e4637795eb21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:34.043527 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:34.043500 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf67e9e-2c20-42bd-869d-e4637795eb21-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw\" (UID: \"3bf67e9e-2c20-42bd-869d-e4637795eb21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:34.111189 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:34.111123 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q54hm\" (UniqueName: \"kubernetes.io/projected/3bf67e9e-2c20-42bd-869d-e4637795eb21-kube-api-access-q54hm\") pod \"payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw\" (UID: \"3bf67e9e-2c20-42bd-869d-e4637795eb21\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:34.180410 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:34.180327 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:34.320329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:34.320286 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc"] Nov 02 20:44:34.557489 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:34.557462 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw"] Nov 02 20:44:34.562941 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:44:34.562899 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf67e9e_2c20_42bd_869d_e4637795eb21.slice/crio-4cb3f78cd61d2b977722b9b0a26343d7027750c1a9ef21b1b2478ea293a0a553 WatchSource:0}: Error finding container 4cb3f78cd61d2b977722b9b0a26343d7027750c1a9ef21b1b2478ea293a0a553: Status 404 returned error can't find the container with id 4cb3f78cd61d2b977722b9b0a26343d7027750c1a9ef21b1b2478ea293a0a553 Nov 02 20:44:35.060361 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.060019 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" event={"ID":"3bf67e9e-2c20-42bd-869d-e4637795eb21","Type":"ContainerStarted","Data":"4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78"} Nov 02 20:44:35.060936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.060386 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" event={"ID":"3bf67e9e-2c20-42bd-869d-e4637795eb21","Type":"ContainerStarted","Data":"4cb3f78cd61d2b977722b9b0a26343d7027750c1a9ef21b1b2478ea293a0a553"} Nov 02 20:44:35.060936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.060420 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:35.064116 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.064087 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" event={"ID":"ef16c845-34db-4717-bae8-173ec711aa55","Type":"ContainerStarted","Data":"b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f"} Nov 02 20:44:35.064270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.064136 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" event={"ID":"ef16c845-34db-4717-bae8-173ec711aa55","Type":"ContainerStarted","Data":"3ce4e3f6d8e4949e5f0a3503b78abb497718f0336b517d2622ac1f94984c7875"} Nov 02 20:44:35.064270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.064191 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:35.110358 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.110330 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86"] Nov 02 20:44:35.119379 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.119349 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2"] Nov 02 20:44:35.156042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.156015 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n"] Nov 02 20:44:35.176177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.169636 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" podStartSLOduration=2.169619094 podStartE2EDuration="2.169619094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:35.168318022 +0000 UTC m=+1747.532929227" watchObservedRunningTime="2026-11-02 20:44:35.169619094 +0000 UTC m=+1747.534230299" Nov 02 20:44:35.278310 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:35.278249 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" podStartSLOduration=2.278232473 podStartE2EDuration="2.278232473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:35.27655561 +0000 UTC m=+1747.641166814" watchObservedRunningTime="2026-11-02 20:44:35.278232473 +0000 UTC m=+1747.642843678" Nov 02 20:44:46.076351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:46.076316 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:44:46.076840 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:46.076556 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:44:48.179614 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:48.179578 5420 generic.go:356] "Generic (PLEG): container finished" podID="37050465-6348-48cb-90bd-36272f806434" containerID="e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d" exitCode=0 Nov 02 20:44:48.180076 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:48.179649 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" event={"ID":"37050465-6348-48cb-90bd-36272f806434","Type":"ContainerDied","Data":"e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d"} Nov 02 20:44:48.180076 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:48.180067 5420 scope.go:122] "RemoveContainer" containerID="e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d" Nov 02 20:44:48.181649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:48.181619 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:44:48.181799 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:48.181710 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:44:48.181799 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:48.181744 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:44:49.198269 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:49.198224 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" event={"ID":"37050465-6348-48cb-90bd-36272f806434","Type":"ContainerStarted","Data":"3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e"} Nov 02 20:44:50.012828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.012798 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:44:50.014129 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.014104 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:44:50.014244 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.014148 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:44:50.131749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.131712 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" containerID="cri-o://c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a" gracePeriod=45 Nov 02 20:44:50.143215 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.143160 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" containerID="cri-o://008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04" gracePeriod=45 Nov 02 20:44:50.185549 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.184880 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" containerID="cri-o://5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad" gracePeriod=45 Nov 02 20:44:50.588930 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.588903 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:44:50.597501 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.593389 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:44:50.605324 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.605297 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:44:50.723346 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723307 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-home\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-home\") pod \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " Nov 02 20:44:50.723514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723368 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-tmp-dir\") pod \"36f9583e-31b6-4da0-b01f-5eace8a27f16\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " Nov 02 20:44:50.723514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723401 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tmp-dir\") pod \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " Nov 02 20:44:50.723514 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723478 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-dshm\") pod \"0a598db0-2671-43ed-8d81-6100f706066d\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " Nov 02 20:44:50.723690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723555 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-kube-api-access-dx688\" (UniqueName: \"kubernetes.io/projected/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-kube-api-access-dx688\") pod \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " Nov 02 20:44:50.723690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723587 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-home\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-home\") pod \"0a598db0-2671-43ed-8d81-6100f706066d\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " Nov 02 20:44:50.723690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723620 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a598db0-2671-43ed-8d81-6100f706066d-kube-api-access-btzfg\" (UniqueName: \"kubernetes.io/projected/0a598db0-2671-43ed-8d81-6100f706066d-kube-api-access-btzfg\") pod \"0a598db0-2671-43ed-8d81-6100f706066d\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " Nov 02 20:44:50.723846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723693 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-dshm\") pod \"36f9583e-31b6-4da0-b01f-5eace8a27f16\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " Nov 02 20:44:50.723846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723726 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-model-cache\") pod \"0a598db0-2671-43ed-8d81-6100f706066d\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " Nov 02 20:44:50.723846 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723782 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a598db0-2671-43ed-8d81-6100f706066d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a598db0-2671-43ed-8d81-6100f706066d-tls-certs\") pod \"0a598db0-2671-43ed-8d81-6100f706066d\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " Nov 02 20:44:50.723992 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723875 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-model-cache\") pod \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " Nov 02 20:44:50.723992 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723931 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tls-certs\") pod \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " Nov 02 20:44:50.724101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.723996 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f9583e-31b6-4da0-b01f-5eace8a27f16-kube-api-access-r59x5\" (UniqueName: \"kubernetes.io/projected/36f9583e-31b6-4da0-b01f-5eace8a27f16-kube-api-access-r59x5\") pod \"36f9583e-31b6-4da0-b01f-5eace8a27f16\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " Nov 02 20:44:50.724101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.724044 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-model-cache\") pod \"36f9583e-31b6-4da0-b01f-5eace8a27f16\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " Nov 02 20:44:50.724101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.724092 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-home\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-home\") pod \"36f9583e-31b6-4da0-b01f-5eace8a27f16\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " Nov 02 20:44:50.724250 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.724159 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f9583e-31b6-4da0-b01f-5eace8a27f16-tls-certs\" (UniqueName: \"kubernetes.io/secret/36f9583e-31b6-4da0-b01f-5eace8a27f16-tls-certs\") pod \"36f9583e-31b6-4da0-b01f-5eace8a27f16\" (UID: \"36f9583e-31b6-4da0-b01f-5eace8a27f16\") " Nov 02 20:44:50.724250 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.724191 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-dshm\") pod \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\" (UID: \"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8\") " Nov 02 20:44:50.724250 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.724218 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-tmp-dir\") pod \"0a598db0-2671-43ed-8d81-6100f706066d\" (UID: \"0a598db0-2671-43ed-8d81-6100f706066d\") " Nov 02 20:44:50.725988 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.725956 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-home" pod "36f9583e-31b6-4da0-b01f-5eace8a27f16" (UID: "36f9583e-31b6-4da0-b01f-5eace8a27f16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.726142 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.726119 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-model-cache" pod "36f9583e-31b6-4da0-b01f-5eace8a27f16" (UID: "36f9583e-31b6-4da0-b01f-5eace8a27f16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.726308 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.726289 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-model-cache" pod "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" (UID: "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.731932 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.731905 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-home" pod "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" (UID: "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.732126 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.732099 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-tmp-dir" pod "36f9583e-31b6-4da0-b01f-5eace8a27f16" (UID: "36f9583e-31b6-4da0-b01f-5eace8a27f16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.732313 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.732294 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tmp-dir" pod "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" (UID: "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.744690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.737923 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-home" pod "0a598db0-2671-43ed-8d81-6100f706066d" (UID: "0a598db0-2671-43ed-8d81-6100f706066d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.744690 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.743041 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-model-cache" pod "0a598db0-2671-43ed-8d81-6100f706066d" (UID: "0a598db0-2671-43ed-8d81-6100f706066d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.748346 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748317 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.748455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748351 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.748455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748375 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.748455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748388 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.748455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748401 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.748455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748412 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.748455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748425 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.748455 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748438 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.749336 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.748642 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-tmp-dir" pod "0a598db0-2671-43ed-8d81-6100f706066d" (UID: "0a598db0-2671-43ed-8d81-6100f706066d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.842621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.841136 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-kube-api-access-dx688" pod "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" (UID: "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8"). InnerVolumeSpecName "kube-api-access-dx688". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:50.842621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.841650 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a598db0-2671-43ed-8d81-6100f706066d-tls-certs" pod "0a598db0-2671-43ed-8d81-6100f706066d" (UID: "0a598db0-2671-43ed-8d81-6100f706066d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:50.845159 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.844968 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tls-certs" pod "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" (UID: "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:50.845159 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.845091 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f9583e-31b6-4da0-b01f-5eace8a27f16-kube-api-access-r59x5" pod "36f9583e-31b6-4da0-b01f-5eace8a27f16" (UID: "36f9583e-31b6-4da0-b01f-5eace8a27f16"). InnerVolumeSpecName "kube-api-access-r59x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:50.845356 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.845165 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-dshm" pod "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" (UID: "4fcc7055-32d1-4eeb-a99f-1a8037dd89f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.845356 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.845314 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f9583e-31b6-4da0-b01f-5eace8a27f16-tls-certs" pod "36f9583e-31b6-4da0-b01f-5eace8a27f16" (UID: "36f9583e-31b6-4da0-b01f-5eace8a27f16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:44:50.845515 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.845496 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-dshm" pod "36f9583e-31b6-4da0-b01f-5eace8a27f16" (UID: "36f9583e-31b6-4da0-b01f-5eace8a27f16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.846480 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.846451 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-dshm" pod "0a598db0-2671-43ed-8d81-6100f706066d" (UID: "0a598db0-2671-43ed-8d81-6100f706066d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:44:50.846851 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.846826 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a598db0-2671-43ed-8d81-6100f706066d-kube-api-access-btzfg" pod "0a598db0-2671-43ed-8d81-6100f706066d" (UID: "0a598db0-2671-43ed-8d81-6100f706066d"). InnerVolumeSpecName "kube-api-access-btzfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:44:50.851249 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.850993 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36f9583e-31b6-4da0-b01f-5eace8a27f16-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851249 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851179 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851249 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851201 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851258 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a598db0-2671-43ed-8d81-6100f706066d-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851279 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx688\" (UniqueName: \"kubernetes.io/projected/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-kube-api-access-dx688\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851292 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btzfg\" (UniqueName: \"kubernetes.io/projected/0a598db0-2671-43ed-8d81-6100f706066d-kube-api-access-btzfg\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851305 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36f9583e-31b6-4da0-b01f-5eace8a27f16-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851339 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a598db0-2671-43ed-8d81-6100f706066d-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851352 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:50.851469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:50.851365 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r59x5\" (UniqueName: \"kubernetes.io/projected/36f9583e-31b6-4da0-b01f-5eace8a27f16-kube-api-access-r59x5\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:44:51.013776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.013421 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:44:51.013776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.013466 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:44:51.092384 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.092346 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:44:51.223232 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.223190 5420 generic.go:356] "Generic (PLEG): container finished" podID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerID="5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad" exitCode=0 Nov 02 20:44:51.223404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.223242 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" Nov 02 20:44:51.223404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.223271 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" event={"ID":"36f9583e-31b6-4da0-b01f-5eace8a27f16","Type":"ContainerDied","Data":"5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad"} Nov 02 20:44:51.223404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.223322 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n" event={"ID":"36f9583e-31b6-4da0-b01f-5eace8a27f16","Type":"ContainerDied","Data":"e4ce7e84ae0a2d5ca768a211a33d845d0af7d9a1364d10c59abe2e3dcd66f566"} Nov 02 20:44:51.223404 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.223346 5420 scope.go:122] "RemoveContainer" containerID="5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad" Nov 02 20:44:51.224880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.224858 5420 generic.go:356] "Generic (PLEG): container finished" podID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerID="008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04" exitCode=0 Nov 02 20:44:51.225005 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.224888 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" event={"ID":"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8","Type":"ContainerDied","Data":"008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04"} Nov 02 20:44:51.225005 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.224932 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" event={"ID":"4fcc7055-32d1-4eeb-a99f-1a8037dd89f8","Type":"ContainerDied","Data":"e9319258dd0fe9dfe208fbb8291e27f411a57bf3b849bc5cf601ce62b399451c"} Nov 02 20:44:51.225359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.225288 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2" Nov 02 20:44:51.231638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.231595 5420 generic.go:356] "Generic (PLEG): container finished" podID="0a598db0-2671-43ed-8d81-6100f706066d" containerID="c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a" exitCode=0 Nov 02 20:44:51.235112 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.231688 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" Nov 02 20:44:51.235112 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.232845 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" event={"ID":"0a598db0-2671-43ed-8d81-6100f706066d","Type":"ContainerDied","Data":"c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a"} Nov 02 20:44:51.235112 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.232906 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" event={"ID":"0a598db0-2671-43ed-8d81-6100f706066d","Type":"ContainerDied","Data":"4b6bee3d238963a98f25cf65f8bc4266da55a2f90b718228912370e424b45a99"} Nov 02 20:44:51.236018 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.236002 5420 scope.go:122] "RemoveContainer" containerID="5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad" Nov 02 20:44:51.236287 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:51.236271 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad\": container with ID starting with 5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad not found: ID does not exist" containerID="5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad" Nov 02 20:44:51.236360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.236293 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad"} err="failed to get container status \"5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad\": rpc error: code = NotFound desc = could not find container \"5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad\": container with ID starting with 5cfd7888635996abb87129b78d4835ffc4d41e826a03acf754c1563d5ae668ad not found: ID does not exist" Nov 02 20:44:51.236360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.236307 5420 scope.go:122] "RemoveContainer" containerID="008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04" Nov 02 20:44:51.245263 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.245248 5420 scope.go:122] "RemoveContainer" containerID="008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04" Nov 02 20:44:51.245508 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:51.245491 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04\": container with ID starting with 008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04 not found: ID does not exist" containerID="008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04" Nov 02 20:44:51.245546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.245516 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04"} err="failed to get container status \"008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04\": rpc error: code = NotFound desc = could not find container \"008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04\": container with ID starting with 008aea6caaf1c115eba9bb6bb089989c8588e22151638a9fa9aa23eba7497e04 not found: ID does not exist" Nov 02 20:44:51.245546 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.245531 5420 scope.go:122] "RemoveContainer" containerID="c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a" Nov 02 20:44:51.254756 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.254738 5420 scope.go:122] "RemoveContainer" containerID="c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a" Nov 02 20:44:51.254983 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:51.254966 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a\": container with ID starting with c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a not found: ID does not exist" containerID="c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a" Nov 02 20:44:51.255027 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.254989 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a"} err="failed to get container status \"c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a\": rpc error: code = NotFound desc = could not find container \"c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a\": container with ID starting with c016c6c2bd06d41889e6f5b8f9db8fb21d4e40771fa838d92c938fcd3481db2a not found: ID does not exist" Nov 02 20:44:51.387084 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.387050 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86"] Nov 02 20:44:51.390645 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.390615 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86"] Nov 02 20:44:51.484257 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.483857 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-c49693/distinct-2-w5-c49693-kserve-6d7c4758ff-tqh86" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:44:51.492946 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.492906 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2"] Nov 02 20:44:51.499801 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.499779 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/distinct-w5-c49693-kserve-5d9fbb9556-fxmj2"] Nov 02 20:44:51.505246 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.505221 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc"] Nov 02 20:44:51.505979 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.505962 5420 cpu_manager.go:414] "containerMap: removing container" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" Nov 02 20:44:51.506042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.505979 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" Nov 02 20:44:51.506042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.505992 5420 cpu_manager.go:414] "containerMap: removing container" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" Nov 02 20:44:51.506042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.505997 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" Nov 02 20:44:51.506042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.506010 5420 cpu_manager.go:414] "containerMap: removing container" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" Nov 02 20:44:51.506042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.506015 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" Nov 02 20:44:51.506188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.506130 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a598db0-2671-43ed-8d81-6100f706066d" containerName="main" Nov 02 20:44:51.506188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.506140 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" containerName="main" Nov 02 20:44:51.506188 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.506146 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" containerName="main" Nov 02 20:44:51.511335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.511316 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.586694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.578063 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-039b89-serving-cert\"" Nov 02 20:44:51.586694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.579959 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.586694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.580160 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2mnt\" (UniqueName: \"kubernetes.io/projected/2e38931c-fe75-457c-b808-03eb5dbb89d8-kube-api-access-n2mnt\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.586694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.580336 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.586694 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.580531 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38931c-fe75-457c-b808-03eb5dbb89d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.590789 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.590764 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc"] Nov 02 20:44:51.600599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.600578 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n"] Nov 02 20:44:51.605206 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.605184 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-c49693/premium-w5-c49693-kserve-dfb67579c-7fm9n"] Nov 02 20:44:51.689767 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.689734 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38931c-fe75-457c-b808-03eb5dbb89d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.689965 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.689945 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.690052 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.689986 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2mnt\" (UniqueName: \"kubernetes.io/projected/2e38931c-fe75-457c-b808-03eb5dbb89d8-kube-api-access-n2mnt\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.690052 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.690039 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.690521 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:51.690500 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-serving-cert: secret "maas-api-e2e-shared-b-w1-039b89-serving-cert" not found Nov 02 20:44:51.690732 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:44:51.690718 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls podName:2e38931c-fe75-457c-b808-03eb5dbb89d8 nodeName:}" failed. No retries permitted until 2026-11-02 20:44:52.190696917 +0000 UTC m=+1764.555308119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls") pod "maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" (UID: "2e38931c-fe75-457c-b808-03eb5dbb89d8") : secret "maas-api-e2e-shared-b-w1-039b89-serving-cert" not found Nov 02 20:44:51.691754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.691730 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38931c-fe75-457c-b808-03eb5dbb89d8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.693122 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.693093 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.808317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.808281 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2mnt\" (UniqueName: \"kubernetes.io/projected/2e38931c-fe75-457c-b808-03eb5dbb89d8-kube-api-access-n2mnt\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:51.810551 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.810511 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a598db0-2671-43ed-8d81-6100f706066d" path="/var/lib/kubelet/pods/0a598db0-2671-43ed-8d81-6100f706066d/volumes" Nov 02 20:44:51.811242 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.811201 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f9583e-31b6-4da0-b01f-5eace8a27f16" path="/var/lib/kubelet/pods/36f9583e-31b6-4da0-b01f-5eace8a27f16/volumes" Nov 02 20:44:51.811979 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:51.811916 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fcc7055-32d1-4eeb-a99f-1a8037dd89f8" path="/var/lib/kubelet/pods/4fcc7055-32d1-4eeb-a99f-1a8037dd89f8/volumes" Nov 02 20:44:52.013241 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:52.013197 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:44:52.013425 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:52.013262 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:44:52.209565 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:52.209479 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:52.211855 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:52.211836 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:52.424724 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:52.424662 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:52.818900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:52.818870 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc"] Nov 02 20:44:52.820038 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:44:52.820008 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e38931c_fe75_457c_b808_03eb5dbb89d8.slice/crio-846273c5dc5a9703ac9b06d47124d5cde10cb7f006d5d2b0675d69e482a6b079 WatchSource:0}: Error finding container 846273c5dc5a9703ac9b06d47124d5cde10cb7f006d5d2b0675d69e482a6b079: Status 404 returned error can't find the container with id 846273c5dc5a9703ac9b06d47124d5cde10cb7f006d5d2b0675d69e482a6b079 Nov 02 20:44:53.013342 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:53.013310 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:44:53.013531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:53.013361 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:44:53.253553 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:53.253463 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" event={"ID":"2e38931c-fe75-457c-b808-03eb5dbb89d8","Type":"ContainerStarted","Data":"846273c5dc5a9703ac9b06d47124d5cde10cb7f006d5d2b0675d69e482a6b079"} Nov 02 20:44:54.013541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:54.013458 5420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 20:44:54.013541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:54.013507 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 20:44:54.283748 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:54.282814 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" event={"ID":"2e38931c-fe75-457c-b808-03eb5dbb89d8","Type":"ContainerStarted","Data":"b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4"} Nov 02 20:44:54.283932 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:54.283902 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:44:54.418232 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:54.418176 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" podStartSLOduration=2.491412835 podStartE2EDuration="3.418161458s" totalImagesPullingTime="926.748623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:44:54.396326407 +0000 UTC m=+1766.760937612" watchObservedRunningTime="2026-11-02 20:44:54.418161458 +0000 UTC m=+1766.782772663" Nov 02 20:44:55.023402 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:55.023363 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:44:55.025314 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:44:55.025290 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:45:00.106125 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:00.106085 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb"] Nov 02 20:45:01.302051 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:01.302021 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:45:02.662022 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:45:02.661990 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:45:15.127329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.127294 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" containerID="cri-o://b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb" gracePeriod=45 Nov 02 20:45:15.388570 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.388516 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:45:15.410998 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.410970 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-model-cache\") pod \"f812f95a-615e-49c3-8362-bcca78368fbc\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " Nov 02 20:45:15.411107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411034 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-dshm\") pod \"f812f95a-615e-49c3-8362-bcca78368fbc\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " Nov 02 20:45:15.411107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411099 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f812f95a-615e-49c3-8362-bcca78368fbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/f812f95a-615e-49c3-8362-bcca78368fbc-tls-certs\") pod \"f812f95a-615e-49c3-8362-bcca78368fbc\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " Nov 02 20:45:15.411177 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411160 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-home\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-home\") pod \"f812f95a-615e-49c3-8362-bcca78368fbc\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " Nov 02 20:45:15.411218 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411204 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-tmp-dir\") pod \"f812f95a-615e-49c3-8362-bcca78368fbc\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " Nov 02 20:45:15.411268 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411250 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f812f95a-615e-49c3-8362-bcca78368fbc-kube-api-access-bf8xj\" (UniqueName: \"kubernetes.io/projected/f812f95a-615e-49c3-8362-bcca78368fbc-kube-api-access-bf8xj\") pod \"f812f95a-615e-49c3-8362-bcca78368fbc\" (UID: \"f812f95a-615e-49c3-8362-bcca78368fbc\") " Nov 02 20:45:15.411268 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411255 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-model-cache" pod "f812f95a-615e-49c3-8362-bcca78368fbc" (UID: "f812f95a-615e-49c3-8362-bcca78368fbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.411469 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411425 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-home" pod "f812f95a-615e-49c3-8362-bcca78368fbc" (UID: "f812f95a-615e-49c3-8362-bcca78368fbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.411535 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411491 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-tmp-dir" pod "f812f95a-615e-49c3-8362-bcca78368fbc" (UID: "f812f95a-615e-49c3-8362-bcca78368fbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.411981 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411959 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.411981 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411981 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.412128 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.411990 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.413253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.413232 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-dshm" pod "f812f95a-615e-49c3-8362-bcca78368fbc" (UID: "f812f95a-615e-49c3-8362-bcca78368fbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:45:15.413253 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.413242 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f812f95a-615e-49c3-8362-bcca78368fbc-tls-certs" pod "f812f95a-615e-49c3-8362-bcca78368fbc" (UID: "f812f95a-615e-49c3-8362-bcca78368fbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:45:15.413367 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.413257 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f812f95a-615e-49c3-8362-bcca78368fbc-kube-api-access-bf8xj" pod "f812f95a-615e-49c3-8362-bcca78368fbc" (UID: "f812f95a-615e-49c3-8362-bcca78368fbc"). InnerVolumeSpecName "kube-api-access-bf8xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:45:15.513066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.513039 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f812f95a-615e-49c3-8362-bcca78368fbc-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.513066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.513068 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf8xj\" (UniqueName: \"kubernetes.io/projected/f812f95a-615e-49c3-8362-bcca78368fbc-kube-api-access-bf8xj\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.513237 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.513080 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f812f95a-615e-49c3-8362-bcca78368fbc-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:45:15.554426 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.554393 5420 generic.go:356] "Generic (PLEG): container finished" podID="f812f95a-615e-49c3-8362-bcca78368fbc" containerID="b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb" exitCode=0 Nov 02 20:45:15.554581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.554434 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" Nov 02 20:45:15.554581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.554477 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" event={"ID":"f812f95a-615e-49c3-8362-bcca78368fbc","Type":"ContainerDied","Data":"b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb"} Nov 02 20:45:15.554581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.554528 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" event={"ID":"f812f95a-615e-49c3-8362-bcca78368fbc","Type":"ContainerDied","Data":"78eacd345de0ea99ee21d957b95c5e4f744b454ed2e3b164c1a120772a3729b8"} Nov 02 20:45:15.554581 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.554550 5420 scope.go:122] "RemoveContainer" containerID="b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb" Nov 02 20:45:15.569028 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.569003 5420 scope.go:122] "RemoveContainer" containerID="b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb" Nov 02 20:45:15.569340 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:45:15.569317 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb\": container with ID starting with b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb not found: ID does not exist" containerID="b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb" Nov 02 20:45:15.569424 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.569348 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb"} err="failed to get container status \"b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb\": rpc error: code = NotFound desc = could not find container \"b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb\": container with ID starting with b4dc6d1ee3e888e98dca4fea77991f9044a8c5ab3d489632ac09ced0ff314cdb not found: ID does not exist" Nov 02 20:45:15.708951 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.708876 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb"] Nov 02 20:45:15.714631 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.714604 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb"] Nov 02 20:45:15.779965 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:15.779932 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" path="/var/lib/kubelet/pods/f812f95a-615e-49c3-8362-bcca78368fbc/volumes" Nov 02 20:45:16.355531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:45:16.355499 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-bce768/premium-w4-bce768-kserve-6dcccf759c-px8lb" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 02 20:45:32.659250 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:45:32.659197 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:46:02.657410 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:46:02.657323 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:46:15.509079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:15.509050 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m"] Nov 02 20:46:15.509499 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:15.509296 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" podUID="13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" containerName="maas-api" containerID="cri-o://004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8" gracePeriod=30 Nov 02 20:46:15.814908 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:15.814881 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:46:16.003955 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.003920 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-metrics-certs\") pod \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " Nov 02 20:46:16.004132 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.004045 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls\") pod \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " Nov 02 20:46:16.004132 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.004099 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-openshift-service-ca\") pod \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " Nov 02 20:46:16.004267 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.004139 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-kube-api-access-ct9sf\" (UniqueName: \"kubernetes.io/projected/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-kube-api-access-ct9sf\") pod \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\" (UID: \"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979\") " Nov 02 20:46:16.004628 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.004591 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-openshift-service-ca" pod "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" (UID: "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:16.004857 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.004836 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.006442 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.006413 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls" pod "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" (UID: "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:16.006526 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.006447 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-kube-api-access-ct9sf" pod "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" (UID: "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979"). InnerVolumeSpecName "kube-api-access-ct9sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:16.006526 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.006497 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-metrics-certs" pod "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" (UID: "13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:16.108792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.108706 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.108792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.108747 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.108792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.108763 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct9sf\" (UniqueName: \"kubernetes.io/projected/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979-kube-api-access-ct9sf\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:16.228133 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.228094 5420 generic.go:356] "Generic (PLEG): container finished" podID="13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" containerID="004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8" exitCode=0 Nov 02 20:46:16.228298 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.228137 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" event={"ID":"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979","Type":"ContainerDied","Data":"004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8"} Nov 02 20:46:16.228298 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.228204 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" event={"ID":"13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979","Type":"ContainerDied","Data":"b0d930e4970f3a70e620f9be568ba59d16e393e44649e7750d18cca1fcd663a2"} Nov 02 20:46:16.228298 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.228234 5420 scope.go:122] "RemoveContainer" containerID="004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8" Nov 02 20:46:16.228443 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.228151 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m" Nov 02 20:46:16.242961 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.242943 5420 scope.go:122] "RemoveContainer" containerID="004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8" Nov 02 20:46:16.250445 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:46:16.250360 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8\": container with ID starting with 004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8 not found: ID does not exist" containerID="004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8" Nov 02 20:46:16.250445 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.250402 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8"} err="failed to get container status \"004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8\": rpc error: code = NotFound desc = could not find container \"004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8\": container with ID starting with 004daeef3c7763643e22966127392b7e27097d72f5d413b6a0de950e726a75d8 not found: ID does not exist" Nov 02 20:46:16.393189 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.393064 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m"] Nov 02 20:46:16.428752 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:16.428718 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-bce768-5699f65dfb-kts5m"] Nov 02 20:46:17.777183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:17.777156 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" path="/var/lib/kubelet/pods/13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979/volumes" Nov 02 20:46:22.137827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.137796 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc"] Nov 02 20:46:22.138527 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.138507 5420 cpu_manager.go:414] "containerMap: removing container" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" Nov 02 20:46:22.138527 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.138526 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" Nov 02 20:46:22.138664 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.138575 5420 cpu_manager.go:414] "containerMap: removing container" podUID="13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" containerName="maas-api" Nov 02 20:46:22.138664 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.138582 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" containerName="maas-api" Nov 02 20:46:22.138761 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.138690 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d7b8cd-4e46-4c8e-bcdf-9e3ffaa9e979" containerName="maas-api" Nov 02 20:46:22.138761 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.138704 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f812f95a-615e-49c3-8362-bcca78368fbc" containerName="main" Nov 02 20:46:22.142494 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.142476 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.189702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.189664 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4n4\" (UniqueName: \"kubernetes.io/projected/b976fa43-0c79-4006-8162-d63a571e96b0-kube-api-access-pv4n4\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.189808 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.189721 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b976fa43-0c79-4006-8162-d63a571e96b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.189880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.189819 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.189978 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.189962 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.193112 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.193086 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-58c2e0-serving-cert\"" Nov 02 20:46:22.242758 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.242731 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc"] Nov 02 20:46:22.291299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.291264 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4n4\" (UniqueName: \"kubernetes.io/projected/b976fa43-0c79-4006-8162-d63a571e96b0-kube-api-access-pv4n4\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.291456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.291319 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b976fa43-0c79-4006-8162-d63a571e96b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.291456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.291370 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.291456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.291454 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.292359 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:46:22.291666 5420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-serving-cert: secret "maas-api-e2e-shared-b-w3-58c2e0-serving-cert" not found Nov 02 20:46:22.292359 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:46:22.291731 5420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls podName:b976fa43-0c79-4006-8162-d63a571e96b0 nodeName:}" failed. No retries permitted until 2026-11-02 20:46:22.791716903 +0000 UTC m=+1855.156328086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls") pod "maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" (UID: "b976fa43-0c79-4006-8162-d63a571e96b0") : secret "maas-api-e2e-shared-b-w3-58c2e0-serving-cert" not found Nov 02 20:46:22.292359 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.292291 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b976fa43-0c79-4006-8162-d63a571e96b0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.295501 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.295447 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.348768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.348740 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4n4\" (UniqueName: \"kubernetes.io/projected/b976fa43-0c79-4006-8162-d63a571e96b0-kube-api-access-pv4n4\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.799732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.799653 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:22.802016 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:22.801985 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:23.056193 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:23.056113 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:23.433926 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:23.433896 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc"] Nov 02 20:46:23.435365 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:46:23.435338 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb976fa43_0c79_4006_8162_d63a571e96b0.slice/crio-57b211a2d8f3d54c118028adc59375133b4ab4d3577c7a5d5ae23a5bb0ac0535 WatchSource:0}: Error finding container 57b211a2d8f3d54c118028adc59375133b4ab4d3577c7a5d5ae23a5bb0ac0535: Status 404 returned error can't find the container with id 57b211a2d8f3d54c118028adc59375133b4ab4d3577c7a5d5ae23a5bb0ac0535 Nov 02 20:46:24.291021 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:24.290992 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" event={"ID":"b976fa43-0c79-4006-8162-d63a571e96b0","Type":"ContainerStarted","Data":"57b211a2d8f3d54c118028adc59375133b4ab4d3577c7a5d5ae23a5bb0ac0535"} Nov 02 20:46:25.301470 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:25.301427 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" event={"ID":"b976fa43-0c79-4006-8162-d63a571e96b0","Type":"ContainerStarted","Data":"68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38"} Nov 02 20:46:25.301884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:25.301501 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:31.315397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:31.315366 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:46:31.426951 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:31.426900 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" podStartSLOduration=8.633797766 podStartE2EDuration="9.426886227s" totalImagesPullingTime="793.088461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:46:25.422495004 +0000 UTC m=+1857.787106249" watchObservedRunningTime="2026-11-02 20:46:31.426886227 +0000 UTC m=+1863.791497477" Nov 02 20:46:32.658710 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:46:32.658480 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:46:42.751885 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:42.751852 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c"] Nov 02 20:46:42.752418 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:42.752093 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" podUID="0afab02f-589b-473a-91d8-35ee552bb40a" containerName="maas-api" containerID="cri-o://6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5" gracePeriod=30 Nov 02 20:46:43.058014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.057990 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:46:43.162732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.162085 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-metrics-certs\") pod \"0afab02f-589b-473a-91d8-35ee552bb40a\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " Nov 02 20:46:43.162732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.162264 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0afab02f-589b-473a-91d8-35ee552bb40a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afab02f-589b-473a-91d8-35ee552bb40a-openshift-service-ca\") pod \"0afab02f-589b-473a-91d8-35ee552bb40a\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " Nov 02 20:46:43.162732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.162323 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0afab02f-589b-473a-91d8-35ee552bb40a-kube-api-access-2rscz\" (UniqueName: \"kubernetes.io/projected/0afab02f-589b-473a-91d8-35ee552bb40a-kube-api-access-2rscz\") pod \"0afab02f-589b-473a-91d8-35ee552bb40a\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " Nov 02 20:46:43.162732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.162435 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls\") pod \"0afab02f-589b-473a-91d8-35ee552bb40a\" (UID: \"0afab02f-589b-473a-91d8-35ee552bb40a\") " Nov 02 20:46:43.165768 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.165732 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afab02f-589b-473a-91d8-35ee552bb40a-openshift-service-ca" pod "0afab02f-589b-473a-91d8-35ee552bb40a" (UID: "0afab02f-589b-473a-91d8-35ee552bb40a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:43.169649 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.169527 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afab02f-589b-473a-91d8-35ee552bb40a-kube-api-access-2rscz" pod "0afab02f-589b-473a-91d8-35ee552bb40a" (UID: "0afab02f-589b-473a-91d8-35ee552bb40a"). InnerVolumeSpecName "kube-api-access-2rscz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:43.174102 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.174051 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-metrics-certs" pod "0afab02f-589b-473a-91d8-35ee552bb40a" (UID: "0afab02f-589b-473a-91d8-35ee552bb40a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:43.176626 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.176595 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls" pod "0afab02f-589b-473a-91d8-35ee552bb40a" (UID: "0afab02f-589b-473a-91d8-35ee552bb40a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:46:43.264528 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.264492 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rscz\" (UniqueName: \"kubernetes.io/projected/0afab02f-589b-473a-91d8-35ee552bb40a-kube-api-access-2rscz\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:43.264528 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.264527 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:43.264528 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.264540 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0afab02f-589b-473a-91d8-35ee552bb40a-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:43.264785 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.264554 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0afab02f-589b-473a-91d8-35ee552bb40a-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:43.468145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.468106 5420 generic.go:356] "Generic (PLEG): container finished" podID="0afab02f-589b-473a-91d8-35ee552bb40a" containerID="6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5" exitCode=0 Nov 02 20:46:43.468326 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.468160 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" Nov 02 20:46:43.468326 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.468186 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" event={"ID":"0afab02f-589b-473a-91d8-35ee552bb40a","Type":"ContainerDied","Data":"6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5"} Nov 02 20:46:43.468326 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.468256 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c" event={"ID":"0afab02f-589b-473a-91d8-35ee552bb40a","Type":"ContainerDied","Data":"bfe9661be2a45b4ef64a3e67b94df1d78c891ceba431fe09032c8e04e65843bf"} Nov 02 20:46:43.468326 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.468286 5420 scope.go:122] "RemoveContainer" containerID="6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5" Nov 02 20:46:43.480718 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.480696 5420 scope.go:122] "RemoveContainer" containerID="6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5" Nov 02 20:46:43.481032 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:46:43.481010 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5\": container with ID starting with 6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5 not found: ID does not exist" containerID="6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5" Nov 02 20:46:43.481100 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.481039 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5"} err="failed to get container status \"6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5\": rpc error: code = NotFound desc = could not find container \"6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5\": container with ID starting with 6e242671185ae612245b28457118179c20804a4e3326e5fc70d20dea6272d0a5 not found: ID does not exist" Nov 02 20:46:43.624728 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.624689 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c"] Nov 02 20:46:43.628175 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.628149 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-654ae4-9566f8c4-2vm2c"] Nov 02 20:46:43.782635 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:43.782523 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0afab02f-589b-473a-91d8-35ee552bb40a" path="/var/lib/kubelet/pods/0afab02f-589b-473a-91d8-35ee552bb40a/volumes" Nov 02 20:46:51.956752 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:51.956720 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44"] Nov 02 20:46:51.957184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:51.956937 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" containerID="cri-o://64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c" gracePeriod=30 Nov 02 20:46:54.515896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:54.515864 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245"] Nov 02 20:46:54.516354 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:54.516082 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" containerID="cri-o://b846640c82272b739f7d719e075740482e2687624b4f2e198d355c1adf5babe0" gracePeriod=30 Nov 02 20:46:54.535650 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:54.535624 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7"] Nov 02 20:46:54.535878 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:54.535839 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" containerID="cri-o://e437a7789a2a95509ce0c368b2cdffdc0cc580132913a43acec91e104d80c6e6" gracePeriod=30 Nov 02 20:46:57.084831 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.084799 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z"] Nov 02 20:46:57.248429 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.248403 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:46:57.373959 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.373914 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-token\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-token\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.374791 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.374767 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-envoy\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.375408 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.375370 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-socket" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:57.375576 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.375052 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-socket\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.375830 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.375812 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-kube-api-access-qp8nz\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-kube-api-access-qp8nz\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.376663 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.376298 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istiod-ca-cert\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.376663 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.376633 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istiod-ca-cert" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:57.377291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.376978 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-certs\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.377291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.377259 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-certs" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:57.377931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.377477 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-data\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.377931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.377712 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-data" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:57.377931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.377902 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-credential-socket" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:57.378270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.378139 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-credential-socket\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.379330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.378637 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-podinfo\") pod \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\" (UID: \"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d\") " Nov 02 20:46:57.379330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.379244 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.379330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.379266 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.379330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.379280 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.379330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.379293 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.379330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.379308 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.380428 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.380361 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-token" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:57.380428 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.380394 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-kube-api-access-qp8nz" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "kube-api-access-qp8nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:57.380662 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.380639 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-envoy" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:57.382276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.382242 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-podinfo" pod "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" (UID: "035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:46:57.480884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.480806 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8nz\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-kube-api-access-qp8nz\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.480884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.480844 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.480884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.480859 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.480884 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.480874 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:57.598535 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.598503 5420 generic.go:356] "Generic (PLEG): container finished" podID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerID="64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c" exitCode=0 Nov 02 20:46:57.598722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.598569 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" Nov 02 20:46:57.598722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.598586 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" event={"ID":"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d","Type":"ContainerDied","Data":"64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c"} Nov 02 20:46:57.598722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.598639 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44" event={"ID":"035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d","Type":"ContainerDied","Data":"2aadaa341cc4fb859eac08ea0831c24e2a1b4155d5811b2c48744560f6487267"} Nov 02 20:46:57.598722 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.598661 5420 scope.go:122] "RemoveContainer" containerID="64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c" Nov 02 20:46:57.610374 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.610354 5420 scope.go:122] "RemoveContainer" containerID="c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea" Nov 02 20:46:57.619264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.619249 5420 scope.go:122] "RemoveContainer" containerID="64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c" Nov 02 20:46:57.619552 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:46:57.619534 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c\": container with ID starting with 64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c not found: ID does not exist" containerID="64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c" Nov 02 20:46:57.619599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.619560 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c"} err="failed to get container status \"64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c\": rpc error: code = NotFound desc = could not find container \"64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c\": container with ID starting with 64adf2f808baa4cffc1c254323bae749b0c97bc984d6b7ff21c65e4b0e0bf77c not found: ID does not exist" Nov 02 20:46:57.619599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.619577 5420 scope.go:122] "RemoveContainer" containerID="c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea" Nov 02 20:46:57.619856 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:46:57.619836 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea\": container with ID starting with c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea not found: ID does not exist" containerID="c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea" Nov 02 20:46:57.619915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.619866 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea"} err="failed to get container status \"c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea\": rpc error: code = NotFound desc = could not find container \"c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea\": container with ID starting with c03bb19630266a52b68a61e2863f3460356897878ba38a72044a10ab9ee176ea not found: ID does not exist" Nov 02 20:46:57.754032 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.753954 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44"] Nov 02 20:46:57.757705 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.757644 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-654ae4-openshift-default-7845c6b779-8jt44"] Nov 02 20:46:57.778351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:57.778313 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" path="/var/lib/kubelet/pods/035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d/volumes" Nov 02 20:46:59.674825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.674568 5420 generic.go:356] "Generic (PLEG): container finished" podID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerID="e437a7789a2a95509ce0c368b2cdffdc0cc580132913a43acec91e104d80c6e6" exitCode=0 Nov 02 20:46:59.674825 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.674758 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" event={"ID":"cd8accdb-1605-45c0-9402-8299d48e0b31","Type":"ContainerDied","Data":"e437a7789a2a95509ce0c368b2cdffdc0cc580132913a43acec91e104d80c6e6"} Nov 02 20:46:59.678084 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.677885 5420 generic.go:356] "Generic (PLEG): container finished" podID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerID="b846640c82272b739f7d719e075740482e2687624b4f2e198d355c1adf5babe0" exitCode=0 Nov 02 20:46:59.678084 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.677986 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" event={"ID":"063bf910-ea3f-4d1c-ac5b-d862a108a343","Type":"ContainerDied","Data":"b846640c82272b739f7d719e075740482e2687624b4f2e198d355c1adf5babe0"} Nov 02 20:46:59.824946 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.824921 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:46:59.870097 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.870065 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:46:59.922742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922718 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-certs\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.922863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922765 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-socket\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.922863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922795 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063bf910-ea3f-4d1c-ac5b-d862a108a343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063bf910-ea3f-4d1c-ac5b-d862a108a343-istiod-ca-cert\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.922863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922816 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-kube-api-access-h9zqk\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-kube-api-access-h9zqk\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.922863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922839 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-token\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-token\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.922863 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922858 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-data\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.923172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922913 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8accdb-1605-45c0-9402-8299d48e0b31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8accdb-1605-45c0-9402-8299d48e0b31-istiod-ca-cert\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.923172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922950 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-envoy\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.923172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.922964 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-certs" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.923172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923016 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-kube-api-access-hj4g8\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-kube-api-access-hj4g8\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.923172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923066 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-certs\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.923172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923138 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-credential-socket\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.923519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923197 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-envoy\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.923519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923209 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-data" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.923519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923226 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-socket\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.923519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923259 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-data\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.923519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923322 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-podinfo\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.923519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923369 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-token\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-token\") pod \"063bf910-ea3f-4d1c-ac5b-d862a108a343\" (UID: \"063bf910-ea3f-4d1c-ac5b-d862a108a343\") " Nov 02 20:46:59.923519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923401 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-credential-socket" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.923519 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923329 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8accdb-1605-45c0-9402-8299d48e0b31-istiod-ca-cert" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:59.923934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923630 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-certs" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.923934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923651 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063bf910-ea3f-4d1c-ac5b-d862a108a343-istiod-ca-cert" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:46:59.923934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923773 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-data" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.923934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923868 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-socket" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.923934 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.923895 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-socket" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.924195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924128 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-podinfo\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.924195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924186 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-credential-socket\") pod \"cd8accdb-1605-45c0-9402-8299d48e0b31\" (UID: \"cd8accdb-1605-45c0-9402-8299d48e0b31\") " Nov 02 20:46:59.924771 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924738 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.924771 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924768 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.924921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924785 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/063bf910-ea3f-4d1c-ac5b-d862a108a343-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.924921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924799 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.924921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924812 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8accdb-1605-45c0-9402-8299d48e0b31-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.924921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924825 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.924921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924838 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.924921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924852 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.924921 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.924865 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:46:59.925296 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.925170 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-credential-socket" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.925521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.925490 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-token" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:59.925619 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.925521 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-kube-api-access-h9zqk" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "kube-api-access-h9zqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:59.925806 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.925788 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-envoy" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:46:59.926005 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.925988 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-podinfo" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:46:59.926005 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.925996 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-kube-api-access-hj4g8" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "kube-api-access-hj4g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:59.926734 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.926713 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-token" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:46:59.926791 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.926722 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-podinfo" pod "cd8accdb-1605-45c0-9402-8299d48e0b31" (UID: "cd8accdb-1605-45c0-9402-8299d48e0b31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:46:59.926791 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:46:59.926767 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-envoy" pod "063bf910-ea3f-4d1c-ac5b-d862a108a343" (UID: "063bf910-ea3f-4d1c-ac5b-d862a108a343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:00.025824 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025791 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.025824 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025819 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.026014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025831 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.026014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025844 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.026014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025857 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9zqk\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-kube-api-access-h9zqk\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.026014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025869 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.026014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025880 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8accdb-1605-45c0-9402-8299d48e0b31-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.026014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025892 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj4g8\" (UniqueName: \"kubernetes.io/projected/063bf910-ea3f-4d1c-ac5b-d862a108a343-kube-api-access-hj4g8\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.026014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.025903 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/063bf910-ea3f-4d1c-ac5b-d862a108a343-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:00.689343 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.689314 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" Nov 02 20:47:00.689810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.689349 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245" event={"ID":"063bf910-ea3f-4d1c-ac5b-d862a108a343","Type":"ContainerDied","Data":"6bc2fee262f5f3ad20f581a0092265f2403d3f411ff6f4b519b44afbfb5436ac"} Nov 02 20:47:00.689810 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.689418 5420 scope.go:122] "RemoveContainer" containerID="b846640c82272b739f7d719e075740482e2687624b4f2e198d355c1adf5babe0" Nov 02 20:47:00.691357 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.691332 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" event={"ID":"cd8accdb-1605-45c0-9402-8299d48e0b31","Type":"ContainerDied","Data":"55b84849d28722edc6d876305bf3349f336ddc594cc10c9c4ea0e11d0987a193"} Nov 02 20:47:00.691549 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.691353 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" Nov 02 20:47:00.703058 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.703014 5420 scope.go:122] "RemoveContainer" containerID="e437a7789a2a95509ce0c368b2cdffdc0cc580132913a43acec91e104d80c6e6" Nov 02 20:47:00.807070 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.807039 5420 patch_prober.go:29] interesting pod/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 20:47:00.807181 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.807089 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 20:47:00.845020 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.844997 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245"] Nov 02 20:47:00.891283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.891261 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-tj245"] Nov 02 20:47:00.994422 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.994375 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7"] Nov 02 20:47:00.998502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:00.998478 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-c49693-openshift-default-854f8968f5-2zcw7"] Nov 02 20:47:01.775082 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:01.775050 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" path="/var/lib/kubelet/pods/063bf910-ea3f-4d1c-ac5b-d862a108a343/volumes" Nov 02 20:47:01.775550 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:01.775521 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" path="/var/lib/kubelet/pods/cd8accdb-1605-45c0-9402-8299d48e0b31/volumes" Nov 02 20:47:02.085486 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.085447 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" podUID="1e74e222-c059-4021-910f-78d9819015c2" containerName="payload-processing" containerID="cri-o://e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717" gracePeriod=25 Nov 02 20:47:02.344063 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.344006 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:47:02.453200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.453172 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e74e222-c059-4021-910f-78d9819015c2-kube-api-access-jqnmg\" (UniqueName: \"kubernetes.io/projected/1e74e222-c059-4021-910f-78d9819015c2-kube-api-access-jqnmg\") pod \"1e74e222-c059-4021-910f-78d9819015c2\" (UID: \"1e74e222-c059-4021-910f-78d9819015c2\") " Nov 02 20:47:02.453348 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.453336 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e74e222-c059-4021-910f-78d9819015c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e74e222-c059-4021-910f-78d9819015c2-plugins-config-volume\") pod \"1e74e222-c059-4021-910f-78d9819015c2\" (UID: \"1e74e222-c059-4021-910f-78d9819015c2\") " Nov 02 20:47:02.455749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.453753 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e74e222-c059-4021-910f-78d9819015c2-plugins-config-volume" pod "1e74e222-c059-4021-910f-78d9819015c2" (UID: "1e74e222-c059-4021-910f-78d9819015c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:02.455749 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.453849 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e74e222-c059-4021-910f-78d9819015c2-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:02.455965 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.455887 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e74e222-c059-4021-910f-78d9819015c2-kube-api-access-jqnmg" pod "1e74e222-c059-4021-910f-78d9819015c2" (UID: "1e74e222-c059-4021-910f-78d9819015c2"). InnerVolumeSpecName "kube-api-access-jqnmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:02.554521 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.554489 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqnmg\" (UniqueName: \"kubernetes.io/projected/1e74e222-c059-4021-910f-78d9819015c2-kube-api-access-jqnmg\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:02.658333 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:02.658253 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:47:02.709730 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.709665 5420 generic.go:356] "Generic (PLEG): container finished" podID="1e74e222-c059-4021-910f-78d9819015c2" containerID="e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717" exitCode=0 Nov 02 20:47:02.709904 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.709747 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" Nov 02 20:47:02.709904 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.709757 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" event={"ID":"1e74e222-c059-4021-910f-78d9819015c2","Type":"ContainerDied","Data":"e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717"} Nov 02 20:47:02.709904 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.709812 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z" event={"ID":"1e74e222-c059-4021-910f-78d9819015c2","Type":"ContainerDied","Data":"c09e788d18f5133a310d0d516e5e0832217456a1e6c2be0e73fca2c426e8ba92"} Nov 02 20:47:02.709904 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.709835 5420 scope.go:122] "RemoveContainer" containerID="e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717" Nov 02 20:47:02.720931 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.720917 5420 scope.go:122] "RemoveContainer" containerID="e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717" Nov 02 20:47:02.721193 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:02.721174 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717\": container with ID starting with e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717 not found: ID does not exist" containerID="e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717" Nov 02 20:47:02.721244 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.721200 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717"} err="failed to get container status \"e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717\": rpc error: code = NotFound desc = could not find container \"e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717\": container with ID starting with e996b80ca76ca849226bf4899637f334f2339eeb56b92d2cb300460d565dd717 not found: ID does not exist" Nov 02 20:47:02.863776 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.863744 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z"] Nov 02 20:47:02.868470 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:02.868444 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-0ea576-dbf4dc7bd-cwd6z"] Nov 02 20:47:03.776485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:03.776453 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e74e222-c059-4021-910f-78d9819015c2" path="/var/lib/kubelet/pods/1e74e222-c059-4021-910f-78d9819015c2/volumes" Nov 02 20:47:04.354138 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:04.354109 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4"] Nov 02 20:47:05.408480 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.408446 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn"] Nov 02 20:47:05.408960 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.408696 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" podUID="3595205a-4595-486b-996d-38dbda6359c1" containerName="maas-api" containerID="cri-o://5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c" gracePeriod=30 Nov 02 20:47:05.687471 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.687448 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:47:05.734619 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.734587 5420 generic.go:356] "Generic (PLEG): container finished" podID="3595205a-4595-486b-996d-38dbda6359c1" containerID="5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c" exitCode=0 Nov 02 20:47:05.734789 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.734627 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" event={"ID":"3595205a-4595-486b-996d-38dbda6359c1","Type":"ContainerDied","Data":"5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c"} Nov 02 20:47:05.734789 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.734694 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" event={"ID":"3595205a-4595-486b-996d-38dbda6359c1","Type":"ContainerDied","Data":"dfc3daaa94f21401fef1bb169031c6e0fad481a187ba43e3536a18d21285e3a8"} Nov 02 20:47:05.734789 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.734630 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn" Nov 02 20:47:05.734918 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.734721 5420 scope.go:122] "RemoveContainer" containerID="5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c" Nov 02 20:47:05.744493 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.744476 5420 scope.go:122] "RemoveContainer" containerID="5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c" Nov 02 20:47:05.744757 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:05.744740 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c\": container with ID starting with 5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c not found: ID does not exist" containerID="5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c" Nov 02 20:47:05.745000 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.744764 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c"} err="failed to get container status \"5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c\": rpc error: code = NotFound desc = could not find container \"5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c\": container with ID starting with 5a153d26f9a8923e5bfb073184014a12761c80b1414166ebfc6377089089a15c not found: ID does not exist" Nov 02 20:47:05.820379 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.820352 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3595205a-4595-486b-996d-38dbda6359c1-kube-api-access-xhnk8\" (UniqueName: \"kubernetes.io/projected/3595205a-4595-486b-996d-38dbda6359c1-kube-api-access-xhnk8\") pod \"3595205a-4595-486b-996d-38dbda6359c1\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " Nov 02 20:47:05.820547 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.820490 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3595205a-4595-486b-996d-38dbda6359c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3595205a-4595-486b-996d-38dbda6359c1-openshift-service-ca\") pod \"3595205a-4595-486b-996d-38dbda6359c1\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " Nov 02 20:47:05.820609 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.820553 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-metrics-certs\") pod \"3595205a-4595-486b-996d-38dbda6359c1\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " Nov 02 20:47:05.820666 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.820618 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls\") pod \"3595205a-4595-486b-996d-38dbda6359c1\" (UID: \"3595205a-4595-486b-996d-38dbda6359c1\") " Nov 02 20:47:05.820914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.820882 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3595205a-4595-486b-996d-38dbda6359c1-openshift-service-ca" pod "3595205a-4595-486b-996d-38dbda6359c1" (UID: "3595205a-4595-486b-996d-38dbda6359c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:05.821166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.821144 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3595205a-4595-486b-996d-38dbda6359c1-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:05.822630 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.822607 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls" pod "3595205a-4595-486b-996d-38dbda6359c1" (UID: "3595205a-4595-486b-996d-38dbda6359c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:47:05.823058 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.823037 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-metrics-certs" pod "3595205a-4595-486b-996d-38dbda6359c1" (UID: "3595205a-4595-486b-996d-38dbda6359c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:47:05.823139 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.823069 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3595205a-4595-486b-996d-38dbda6359c1-kube-api-access-xhnk8" pod "3595205a-4595-486b-996d-38dbda6359c1" (UID: "3595205a-4595-486b-996d-38dbda6359c1"). InnerVolumeSpecName "kube-api-access-xhnk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:05.941301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.935816 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhnk8\" (UniqueName: \"kubernetes.io/projected/3595205a-4595-486b-996d-38dbda6359c1-kube-api-access-xhnk8\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:05.941301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.935858 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:05.941301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:05.935872 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3595205a-4595-486b-996d-38dbda6359c1-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:06.157823 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:06.157780 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn"] Nov 02 20:47:06.163365 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:06.163334 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-0ea576-774db8cd4-g2qqn"] Nov 02 20:47:07.776765 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:07.776737 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3595205a-4595-486b-996d-38dbda6359c1" path="/var/lib/kubelet/pods/3595205a-4595-486b-996d-38dbda6359c1/volumes" Nov 02 20:47:09.355408 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.355302 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" podUID="f4761389-0a5c-4fbf-a988-3a5c7e45c80d" containerName="payload-processing" containerID="cri-o://c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94" gracePeriod=25 Nov 02 20:47:09.613793 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.613740 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:47:09.677565 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.677539 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-kube-api-access-s2pv7\" (UniqueName: \"kubernetes.io/projected/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-kube-api-access-s2pv7\") pod \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\" (UID: \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\") " Nov 02 20:47:09.677747 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.677688 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-plugins-config-volume\") pod \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\" (UID: \"f4761389-0a5c-4fbf-a988-3a5c7e45c80d\") " Nov 02 20:47:09.678019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.678000 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-plugins-config-volume" pod "f4761389-0a5c-4fbf-a988-3a5c7e45c80d" (UID: "f4761389-0a5c-4fbf-a988-3a5c7e45c80d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:09.679532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.679511 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-kube-api-access-s2pv7" pod "f4761389-0a5c-4fbf-a988-3a5c7e45c80d" (UID: "f4761389-0a5c-4fbf-a988-3a5c7e45c80d"). InnerVolumeSpecName "kube-api-access-s2pv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:09.766074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.765881 5420 generic.go:356] "Generic (PLEG): container finished" podID="f4761389-0a5c-4fbf-a988-3a5c7e45c80d" containerID="c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94" exitCode=0 Nov 02 20:47:09.766074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.765932 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" Nov 02 20:47:09.766074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.765971 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" event={"ID":"f4761389-0a5c-4fbf-a988-3a5c7e45c80d","Type":"ContainerDied","Data":"c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94"} Nov 02 20:47:09.766074 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.766059 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4" event={"ID":"f4761389-0a5c-4fbf-a988-3a5c7e45c80d","Type":"ContainerDied","Data":"8f39652f9a540c8dfbe5c69e1eabccaaf2de46eb5f5979a8f291c274ef37f599"} Nov 02 20:47:09.766380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.766087 5420 scope.go:122] "RemoveContainer" containerID="c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94" Nov 02 20:47:09.778924 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.778901 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:09.779037 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.778927 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2pv7\" (UniqueName: \"kubernetes.io/projected/f4761389-0a5c-4fbf-a988-3a5c7e45c80d-kube-api-access-s2pv7\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:09.780174 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.780152 5420 scope.go:122] "RemoveContainer" containerID="c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94" Nov 02 20:47:09.780424 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:09.780405 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94\": container with ID starting with c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94 not found: ID does not exist" containerID="c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94" Nov 02 20:47:09.780488 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.780433 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94"} err="failed to get container status \"c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94\": rpc error: code = NotFound desc = could not find container \"c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94\": container with ID starting with c3df38d4cbe48afb79cb835fc10bad0c14463f2645419b210060324ae2dc5d94 not found: ID does not exist" Nov 02 20:47:09.919306 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.919237 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4"] Nov 02 20:47:09.934507 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:09.934481 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-039b89-7477df79bb-smhq4"] Nov 02 20:47:11.778382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:11.778345 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4761389-0a5c-4fbf-a988-3a5c7e45c80d" path="/var/lib/kubelet/pods/f4761389-0a5c-4fbf-a988-3a5c7e45c80d/volumes" Nov 02 20:47:14.180278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:14.180249 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2"] Nov 02 20:47:14.181472 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:14.180538 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" containerID="cri-o://5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651" gracePeriod=30 Nov 02 20:47:19.457226 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.457172 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:47:19.612901 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.612877 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-token\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-token\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613060 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.612930 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-certs\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613060 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.612949 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-socket\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613060 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.612969 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-kube-api-access-h8nkv\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613060 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613025 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-podinfo\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613060 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613055 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istiod-ca-cert\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613333 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613086 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-data\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613333 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613183 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-credential-socket\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613333 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613213 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-envoy\") pod \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\" (UID: \"eaa56fb7-abaa-4169-9c0a-319feee7dc86\") " Nov 02 20:47:19.613333 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613241 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-socket" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:19.613333 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613254 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-certs" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:19.613583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613438 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-data" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:19.613583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613475 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-credential-socket" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:19.613583 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613494 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istiod-ca-cert" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:19.613859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613838 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.613927 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613863 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.613927 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613878 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.613927 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613891 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.613927 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.613904 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.615298 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.615269 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-envoy" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:19.615298 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.615288 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-kube-api-access-h8nkv" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "kube-api-access-h8nkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:19.615429 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.615330 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-podinfo" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:47:19.615466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.615418 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-token" pod "eaa56fb7-abaa-4169-9c0a-319feee7dc86" (UID: "eaa56fb7-abaa-4169-9c0a-319feee7dc86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:19.714595 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.714532 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.714595 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.714560 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8nkv\" (UniqueName: \"kubernetes.io/projected/eaa56fb7-abaa-4169-9c0a-319feee7dc86-kube-api-access-h8nkv\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.714595 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.714573 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.714595 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.714585 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa56fb7-abaa-4169-9c0a-319feee7dc86-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:19.840929 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.840902 5420 generic.go:356] "Generic (PLEG): container finished" podID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerID="5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651" exitCode=0 Nov 02 20:47:19.841049 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.840949 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" Nov 02 20:47:19.841049 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.840984 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" event={"ID":"eaa56fb7-abaa-4169-9c0a-319feee7dc86","Type":"ContainerDied","Data":"5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651"} Nov 02 20:47:19.841049 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.841034 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2" event={"ID":"eaa56fb7-abaa-4169-9c0a-319feee7dc86","Type":"ContainerDied","Data":"334c8784bc5b7f168c481d9892de98444861bb7da852cec97a530b0076013ab4"} Nov 02 20:47:19.841162 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.841056 5420 scope.go:122] "RemoveContainer" containerID="5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651" Nov 02 20:47:19.851110 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.851096 5420 scope.go:122] "RemoveContainer" containerID="a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc" Nov 02 20:47:19.860497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.860473 5420 scope.go:122] "RemoveContainer" containerID="5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651" Nov 02 20:47:19.860756 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:19.860737 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651\": container with ID starting with 5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651 not found: ID does not exist" containerID="5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651" Nov 02 20:47:19.860844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.860761 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651"} err="failed to get container status \"5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651\": rpc error: code = NotFound desc = could not find container \"5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651\": container with ID starting with 5731f9ba5de3b8f56dd11f826dcaf1ebccda5b11e55a42a6477f04dc54765651 not found: ID does not exist" Nov 02 20:47:19.860844 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.860776 5420 scope.go:122] "RemoveContainer" containerID="a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc" Nov 02 20:47:19.861031 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:19.861016 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc\": container with ID starting with a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc not found: ID does not exist" containerID="a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc" Nov 02 20:47:19.861089 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.861038 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc"} err="failed to get container status \"a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc\": rpc error: code = NotFound desc = could not find container \"a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc\": container with ID starting with a5ffe32e9c56ec5d86a34f39f452728fb329fa3caf6bdb6bc0e0049d168038bc not found: ID does not exist" Nov 02 20:47:19.996638 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:19.996569 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2"] Nov 02 20:47:20.001013 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:20.000986 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0ea576-openshift-default-5f757f8587-cq2d2"] Nov 02 20:47:21.776325 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:21.776290 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" path="/var/lib/kubelet/pods/eaa56fb7-abaa-4169-9c0a-319feee7dc86/volumes" Nov 02 20:47:25.065147 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:25.065085 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf"] Nov 02 20:47:25.065656 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:25.065409 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" containerID="cri-o://f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad" gracePeriod=30 Nov 02 20:47:29.489684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:29.489601 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw"] Nov 02 20:47:29.599916 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:29.599849 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc"] Nov 02 20:47:30.346029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.345966 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.356943 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-credential-socket\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.356991 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-certs\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.357057 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a9bf42-6667-4ed6-b824-b4b4f209f989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a9bf42-6667-4ed6-b824-b4b4f209f989-istiod-ca-cert\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.357108 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-podinfo\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.357173 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-data\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.357204 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-socket\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.357396 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-kube-api-access-cbp65\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-kube-api-access-cbp65\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.357429 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-token\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-token\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.357842 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.357457 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-envoy\") pod \"01a9bf42-6667-4ed6-b824-b4b4f209f989\" (UID: \"01a9bf42-6667-4ed6-b824-b4b4f209f989\") " Nov 02 20:47:30.361730 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.361696 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-certs" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:30.361860 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.361785 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-credential-socket" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:30.361996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.361956 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-data" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:30.361996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.361965 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a9bf42-6667-4ed6-b824-b4b4f209f989-istiod-ca-cert" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:30.362603 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.362571 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-socket" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:30.363924 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.363898 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-podinfo" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:47:30.364240 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.364220 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-token" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:30.364322 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.364244 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-envoy" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:47:30.364322 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.364310 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-kube-api-access-cbp65" pod "01a9bf42-6667-4ed6-b824-b4b4f209f989" (UID: "01a9bf42-6667-4ed6-b824-b4b4f209f989"). InnerVolumeSpecName "kube-api-access-cbp65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:30.458889 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458865 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbp65\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-kube-api-access-cbp65\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.458889 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458888 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.459035 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458936 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.459035 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458948 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.459035 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458958 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.459035 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458966 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a9bf42-6667-4ed6-b824-b4b4f209f989-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.459035 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458974 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.459035 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458982 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.459035 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.458989 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a9bf42-6667-4ed6-b824-b4b4f209f989-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:30.973997 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.973956 5420 generic.go:356] "Generic (PLEG): container finished" podID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerID="f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad" exitCode=0 Nov 02 20:47:30.974502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.974037 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" Nov 02 20:47:30.974502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.974201 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" event={"ID":"01a9bf42-6667-4ed6-b824-b4b4f209f989","Type":"ContainerDied","Data":"f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad"} Nov 02 20:47:30.974502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.974255 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf" event={"ID":"01a9bf42-6667-4ed6-b824-b4b4f209f989","Type":"ContainerDied","Data":"4a9c93fa3b382c9edb2d643050065aae29ec060830c51776799764dece142763"} Nov 02 20:47:30.974502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:30.974284 5420 scope.go:122] "RemoveContainer" containerID="f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad" Nov 02 20:47:31.035502 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:31.035471 5420 scope.go:122] "RemoveContainer" containerID="f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad" Nov 02 20:47:31.036096 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:31.036060 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad\": container with ID starting with f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad not found: ID does not exist" containerID="f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad" Nov 02 20:47:31.036217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:31.036112 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad"} err="failed to get container status \"f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad\": rpc error: code = NotFound desc = could not find container \"f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad\": container with ID starting with f126783cc22e19daaae3db942de288b9d23e35290c459b0bdc7e06891d88c8ad not found: ID does not exist" Nov 02 20:47:31.129152 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:31.129118 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf"] Nov 02 20:47:31.177050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:31.177017 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-6jhlf"] Nov 02 20:47:31.777792 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:31.777758 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" path="/var/lib/kubelet/pods/01a9bf42-6667-4ed6-b824-b4b4f209f989/volumes" Nov 02 20:47:32.659192 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:32.659151 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:47:34.490425 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.490384 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" podUID="3bf67e9e-2c20-42bd-869d-e4637795eb21" containerName="payload-processing" containerID="cri-o://4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78" gracePeriod=25 Nov 02 20:47:34.600773 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.600722 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" podUID="ef16c845-34db-4717-bae8-173ec711aa55" containerName="payload-pre-processing" containerID="cri-o://b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f" gracePeriod=25 Nov 02 20:47:34.781358 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.781016 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:47:34.814577 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.813608 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf67e9e-2c20-42bd-869d-e4637795eb21-kube-api-access-q54hm\" (UniqueName: \"kubernetes.io/projected/3bf67e9e-2c20-42bd-869d-e4637795eb21-kube-api-access-q54hm\") pod \"3bf67e9e-2c20-42bd-869d-e4637795eb21\" (UID: \"3bf67e9e-2c20-42bd-869d-e4637795eb21\") " Nov 02 20:47:34.814577 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.813688 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf67e9e-2c20-42bd-869d-e4637795eb21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf67e9e-2c20-42bd-869d-e4637795eb21-plugins-config-volume\") pod \"3bf67e9e-2c20-42bd-869d-e4637795eb21\" (UID: \"3bf67e9e-2c20-42bd-869d-e4637795eb21\") " Nov 02 20:47:34.815301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.815275 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf67e9e-2c20-42bd-869d-e4637795eb21-plugins-config-volume" pod "3bf67e9e-2c20-42bd-869d-e4637795eb21" (UID: "3bf67e9e-2c20-42bd-869d-e4637795eb21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:34.816802 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.816241 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf67e9e-2c20-42bd-869d-e4637795eb21-kube-api-access-q54hm" pod "3bf67e9e-2c20-42bd-869d-e4637795eb21" (UID: "3bf67e9e-2c20-42bd-869d-e4637795eb21"). InnerVolumeSpecName "kube-api-access-q54hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:34.868692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.868657 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:47:34.915119 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.915095 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef16c845-34db-4717-bae8-173ec711aa55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef16c845-34db-4717-bae8-173ec711aa55-plugins-config-volume\") pod \"ef16c845-34db-4717-bae8-173ec711aa55\" (UID: \"ef16c845-34db-4717-bae8-173ec711aa55\") " Nov 02 20:47:34.915236 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.915165 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef16c845-34db-4717-bae8-173ec711aa55-kube-api-access-rrf8f\" (UniqueName: \"kubernetes.io/projected/ef16c845-34db-4717-bae8-173ec711aa55-kube-api-access-rrf8f\") pod \"ef16c845-34db-4717-bae8-173ec711aa55\" (UID: \"ef16c845-34db-4717-bae8-173ec711aa55\") " Nov 02 20:47:34.915520 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.915501 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bf67e9e-2c20-42bd-869d-e4637795eb21-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:34.915520 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.915521 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q54hm\" (UniqueName: \"kubernetes.io/projected/3bf67e9e-2c20-42bd-869d-e4637795eb21-kube-api-access-q54hm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:34.915735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.915547 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef16c845-34db-4717-bae8-173ec711aa55-plugins-config-volume" pod "ef16c845-34db-4717-bae8-173ec711aa55" (UID: "ef16c845-34db-4717-bae8-173ec711aa55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:34.917342 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:34.917324 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef16c845-34db-4717-bae8-173ec711aa55-kube-api-access-rrf8f" pod "ef16c845-34db-4717-bae8-173ec711aa55" (UID: "ef16c845-34db-4717-bae8-173ec711aa55"). InnerVolumeSpecName "kube-api-access-rrf8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:35.017023 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.016949 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrf8f\" (UniqueName: \"kubernetes.io/projected/ef16c845-34db-4717-bae8-173ec711aa55-kube-api-access-rrf8f\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:35.017237 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.017220 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef16c845-34db-4717-bae8-173ec711aa55-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:35.022191 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.022159 5420 generic.go:356] "Generic (PLEG): container finished" podID="ef16c845-34db-4717-bae8-173ec711aa55" containerID="b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f" exitCode=0 Nov 02 20:47:35.022371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.022200 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" event={"ID":"ef16c845-34db-4717-bae8-173ec711aa55","Type":"ContainerDied","Data":"b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f"} Nov 02 20:47:35.022479 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.022217 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" Nov 02 20:47:35.022479 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.022429 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc" event={"ID":"ef16c845-34db-4717-bae8-173ec711aa55","Type":"ContainerDied","Data":"3ce4e3f6d8e4949e5f0a3503b78abb497718f0336b517d2622ac1f94984c7875"} Nov 02 20:47:35.022479 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.022478 5420 scope.go:122] "RemoveContainer" containerID="b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f" Nov 02 20:47:35.025390 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.025365 5420 generic.go:356] "Generic (PLEG): container finished" podID="3bf67e9e-2c20-42bd-869d-e4637795eb21" containerID="4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78" exitCode=0 Nov 02 20:47:35.025497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.025438 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" Nov 02 20:47:35.025497 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.025448 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" event={"ID":"3bf67e9e-2c20-42bd-869d-e4637795eb21","Type":"ContainerDied","Data":"4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78"} Nov 02 20:47:35.025601 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.025501 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw" event={"ID":"3bf67e9e-2c20-42bd-869d-e4637795eb21","Type":"ContainerDied","Data":"4cb3f78cd61d2b977722b9b0a26343d7027750c1a9ef21b1b2478ea293a0a553"} Nov 02 20:47:35.036254 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.036234 5420 scope.go:122] "RemoveContainer" containerID="b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f" Nov 02 20:47:35.036538 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:35.036515 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f\": container with ID starting with b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f not found: ID does not exist" containerID="b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f" Nov 02 20:47:35.036600 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.036549 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f"} err="failed to get container status \"b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f\": rpc error: code = NotFound desc = could not find container \"b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f\": container with ID starting with b8b6051baae181a4533ca83dc2e9cff8ddde318483926fcd0a1103bc5a30095f not found: ID does not exist" Nov 02 20:47:35.036600 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.036571 5420 scope.go:122] "RemoveContainer" containerID="4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78" Nov 02 20:47:35.047928 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.047910 5420 scope.go:122] "RemoveContainer" containerID="4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78" Nov 02 20:47:35.048186 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:35.048162 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78\": container with ID starting with 4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78 not found: ID does not exist" containerID="4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78" Nov 02 20:47:35.048266 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.048190 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78"} err="failed to get container status \"4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78\": rpc error: code = NotFound desc = could not find container \"4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78\": container with ID starting with 4dc6bf18142048ff032584f8b8fd29107bca7c8a290f245d653d9640dc225f78 not found: ID does not exist" Nov 02 20:47:35.178069 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.178038 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc"] Nov 02 20:47:35.184100 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.184074 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-18fd4c-596c75dcf9-khntc"] Nov 02 20:47:35.283841 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.283759 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw"] Nov 02 20:47:35.289783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.289756 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-18fd4c-55df88d8fd-ql8nw"] Nov 02 20:47:35.777150 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.777117 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf67e9e-2c20-42bd-869d-e4637795eb21" path="/var/lib/kubelet/pods/3bf67e9e-2c20-42bd-869d-e4637795eb21/volumes" Nov 02 20:47:35.777637 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:35.777617 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef16c845-34db-4717-bae8-173ec711aa55" path="/var/lib/kubelet/pods/ef16c845-34db-4717-bae8-173ec711aa55/volumes" Nov 02 20:47:48.761349 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:48.761309 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc"] Nov 02 20:47:48.761948 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:48.761566 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" podUID="2e38931c-fe75-457c-b808-03eb5dbb89d8" containerName="maas-api" containerID="cri-o://b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4" gracePeriod=30 Nov 02 20:47:49.054853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.054827 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:47:49.123330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.123300 5420 generic.go:356] "Generic (PLEG): container finished" podID="2e38931c-fe75-457c-b808-03eb5dbb89d8" containerID="b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4" exitCode=0 Nov 02 20:47:49.123471 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.123350 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" Nov 02 20:47:49.123471 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.123381 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" event={"ID":"2e38931c-fe75-457c-b808-03eb5dbb89d8","Type":"ContainerDied","Data":"b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4"} Nov 02 20:47:49.123471 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.123431 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc" event={"ID":"2e38931c-fe75-457c-b808-03eb5dbb89d8","Type":"ContainerDied","Data":"846273c5dc5a9703ac9b06d47124d5cde10cb7f006d5d2b0675d69e482a6b079"} Nov 02 20:47:49.123471 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.123452 5420 scope.go:122] "RemoveContainer" containerID="b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4" Nov 02 20:47:49.136619 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.136602 5420 scope.go:122] "RemoveContainer" containerID="b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4" Nov 02 20:47:49.136909 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:47:49.136887 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4\": container with ID starting with b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4 not found: ID does not exist" containerID="b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4" Nov 02 20:47:49.136986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.136918 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4"} err="failed to get container status \"b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4\": rpc error: code = NotFound desc = could not find container \"b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4\": container with ID starting with b33c96c5a4684753f5efc090e21f4468f268e3a6cef114c866ba1f1df1fd38c4 not found: ID does not exist" Nov 02 20:47:49.194056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.192967 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e38931c-fe75-457c-b808-03eb5dbb89d8-kube-api-access-n2mnt\" (UniqueName: \"kubernetes.io/projected/2e38931c-fe75-457c-b808-03eb5dbb89d8-kube-api-access-n2mnt\") pod \"2e38931c-fe75-457c-b808-03eb5dbb89d8\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " Nov 02 20:47:49.194056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.193117 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e38931c-fe75-457c-b808-03eb5dbb89d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38931c-fe75-457c-b808-03eb5dbb89d8-openshift-service-ca\") pod \"2e38931c-fe75-457c-b808-03eb5dbb89d8\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " Nov 02 20:47:49.194056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.193224 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls\") pod \"2e38931c-fe75-457c-b808-03eb5dbb89d8\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " Nov 02 20:47:49.194056 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.193313 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-metrics-certs\") pod \"2e38931c-fe75-457c-b808-03eb5dbb89d8\" (UID: \"2e38931c-fe75-457c-b808-03eb5dbb89d8\") " Nov 02 20:47:49.219545 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.211021 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e38931c-fe75-457c-b808-03eb5dbb89d8-openshift-service-ca" pod "2e38931c-fe75-457c-b808-03eb5dbb89d8" (UID: "2e38931c-fe75-457c-b808-03eb5dbb89d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:47:49.219545 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.218882 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e38931c-fe75-457c-b808-03eb5dbb89d8-kube-api-access-n2mnt" pod "2e38931c-fe75-457c-b808-03eb5dbb89d8" (UID: "2e38931c-fe75-457c-b808-03eb5dbb89d8"). InnerVolumeSpecName "kube-api-access-n2mnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:47:49.220115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.220076 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-metrics-certs" pod "2e38931c-fe75-457c-b808-03eb5dbb89d8" (UID: "2e38931c-fe75-457c-b808-03eb5dbb89d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:47:49.220554 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.220532 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls" pod "2e38931c-fe75-457c-b808-03eb5dbb89d8" (UID: "2e38931c-fe75-457c-b808-03eb5dbb89d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:47:49.297754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.297202 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:49.297754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.297238 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e38931c-fe75-457c-b808-03eb5dbb89d8-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:49.297754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.297272 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2mnt\" (UniqueName: \"kubernetes.io/projected/2e38931c-fe75-457c-b808-03eb5dbb89d8-kube-api-access-n2mnt\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:49.297754 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.297290 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e38931c-fe75-457c-b808-03eb5dbb89d8-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:47:49.536896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.536863 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc"] Nov 02 20:47:49.540648 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.540619 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-039b89-6cc7895685-h5csc"] Nov 02 20:47:49.789180 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:49.789149 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e38931c-fe75-457c-b808-03eb5dbb89d8" path="/var/lib/kubelet/pods/2e38931c-fe75-457c-b808-03eb5dbb89d8/volumes" Nov 02 20:47:52.800663 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:52.800632 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7"] Nov 02 20:47:58.158466 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:47:58.158416 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" podUID="733a96e9-08bd-48d8-a015-aac03b32ffd6" containerName="payload-processing" containerID="cri-o://d3c43470cda3b4879063950db952d63d6123a8e5a0f3c5bfe27f7c8ac03c4fb3" gracePeriod=25 Nov 02 20:48:02.657758 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:02.657721 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:48:03.295059 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:03.295025 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm"] Nov 02 20:48:04.330885 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:04.330849 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" podUID="733a96e9-08bd-48d8-a015-aac03b32ffd6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:48:05.814585 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:05.814549 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv"] Nov 02 20:48:05.814982 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:05.814763 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" podUID="90ab3a61-5a80-4946-841b-c937011affc4" containerName="maas-api" containerID="cri-o://f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb" gracePeriod=30 Nov 02 20:48:06.116830 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.116803 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:48:06.224272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.223272 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-metrics-certs\") pod \"90ab3a61-5a80-4946-841b-c937011affc4\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " Nov 02 20:48:06.224272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.223435 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls\") pod \"90ab3a61-5a80-4946-841b-c937011affc4\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " Nov 02 20:48:06.224272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.223501 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ab3a61-5a80-4946-841b-c937011affc4-kube-api-access-v6zr5\" (UniqueName: \"kubernetes.io/projected/90ab3a61-5a80-4946-841b-c937011affc4-kube-api-access-v6zr5\") pod \"90ab3a61-5a80-4946-841b-c937011affc4\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " Nov 02 20:48:06.224272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.223583 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ab3a61-5a80-4946-841b-c937011affc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ab3a61-5a80-4946-841b-c937011affc4-openshift-service-ca\") pod \"90ab3a61-5a80-4946-841b-c937011affc4\" (UID: \"90ab3a61-5a80-4946-841b-c937011affc4\") " Nov 02 20:48:06.225136 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.224805 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ab3a61-5a80-4946-841b-c937011affc4-openshift-service-ca" pod "90ab3a61-5a80-4946-841b-c937011affc4" (UID: "90ab3a61-5a80-4946-841b-c937011affc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:06.242073 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.240704 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ab3a61-5a80-4946-841b-c937011affc4-kube-api-access-v6zr5" pod "90ab3a61-5a80-4946-841b-c937011affc4" (UID: "90ab3a61-5a80-4946-841b-c937011affc4"). InnerVolumeSpecName "kube-api-access-v6zr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:06.244301 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.244258 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-metrics-certs" pod "90ab3a61-5a80-4946-841b-c937011affc4" (UID: "90ab3a61-5a80-4946-841b-c937011affc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:06.246227 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.246200 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls" pod "90ab3a61-5a80-4946-841b-c937011affc4" (UID: "90ab3a61-5a80-4946-841b-c937011affc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:06.268757 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.268728 5420 generic.go:356] "Generic (PLEG): container finished" podID="90ab3a61-5a80-4946-841b-c937011affc4" containerID="f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb" exitCode=0 Nov 02 20:48:06.268857 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.268829 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" event={"ID":"90ab3a61-5a80-4946-841b-c937011affc4","Type":"ContainerDied","Data":"f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb"} Nov 02 20:48:06.268911 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.268882 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" event={"ID":"90ab3a61-5a80-4946-841b-c937011affc4","Type":"ContainerDied","Data":"d96d458ea2e1ecfbefdea40ebac383a39bef473904d1aa42abd523086ba53a3e"} Nov 02 20:48:06.268949 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.268909 5420 scope.go:122] "RemoveContainer" containerID="f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb" Nov 02 20:48:06.269068 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.269056 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv" Nov 02 20:48:06.284861 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.284839 5420 scope.go:122] "RemoveContainer" containerID="f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb" Nov 02 20:48:06.285160 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:06.285132 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb\": container with ID starting with f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb not found: ID does not exist" containerID="f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb" Nov 02 20:48:06.285239 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.285170 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb"} err="failed to get container status \"f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb\": rpc error: code = NotFound desc = could not find container \"f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb\": container with ID starting with f94f425d74a1d7cde865e7d07c1f0bca1fd39b1ec32785e832b31f8439bf6edb not found: ID does not exist" Nov 02 20:48:06.325161 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.325138 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:06.325161 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.325158 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6zr5\" (UniqueName: \"kubernetes.io/projected/90ab3a61-5a80-4946-841b-c937011affc4-kube-api-access-v6zr5\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:06.325278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.325169 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ab3a61-5a80-4946-841b-c937011affc4-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:06.325278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.325178 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90ab3a61-5a80-4946-841b-c937011affc4-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:06.424974 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.424882 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv"] Nov 02 20:48:06.428908 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:06.428875 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-1620dc-68cdf86bfd-zv7pv"] Nov 02 20:48:07.780738 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:07.780707 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ab3a61-5a80-4946-841b-c937011affc4" path="/var/lib/kubelet/pods/90ab3a61-5a80-4946-841b-c937011affc4/volumes" Nov 02 20:48:14.331069 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:14.331034 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" podUID="733a96e9-08bd-48d8-a015-aac03b32ffd6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:48:18.318440 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.318372 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" containerID="cri-o://8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0" gracePeriod=45 Nov 02 20:48:18.346563 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.346528 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": dial tcp 10.133.0.69:8000: connect: connection refused" Nov 02 20:48:18.361189 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.361161 5420 generic.go:356] "Generic (PLEG): container finished" podID="733a96e9-08bd-48d8-a015-aac03b32ffd6" containerID="d3c43470cda3b4879063950db952d63d6123a8e5a0f3c5bfe27f7c8ac03c4fb3" exitCode=0 Nov 02 20:48:18.361309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.361216 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" event={"ID":"733a96e9-08bd-48d8-a015-aac03b32ffd6","Type":"ContainerDied","Data":"d3c43470cda3b4879063950db952d63d6123a8e5a0f3c5bfe27f7c8ac03c4fb3"} Nov 02 20:48:18.456970 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.456939 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:48:18.555266 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.555238 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733a96e9-08bd-48d8-a015-aac03b32ffd6-kube-api-access-6vdcn\" (UniqueName: \"kubernetes.io/projected/733a96e9-08bd-48d8-a015-aac03b32ffd6-kube-api-access-6vdcn\") pod \"733a96e9-08bd-48d8-a015-aac03b32ffd6\" (UID: \"733a96e9-08bd-48d8-a015-aac03b32ffd6\") " Nov 02 20:48:18.555414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.555396 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733a96e9-08bd-48d8-a015-aac03b32ffd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733a96e9-08bd-48d8-a015-aac03b32ffd6-plugins-config-volume\") pod \"733a96e9-08bd-48d8-a015-aac03b32ffd6\" (UID: \"733a96e9-08bd-48d8-a015-aac03b32ffd6\") " Nov 02 20:48:18.555881 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.555853 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733a96e9-08bd-48d8-a015-aac03b32ffd6-plugins-config-volume" pod "733a96e9-08bd-48d8-a015-aac03b32ffd6" (UID: "733a96e9-08bd-48d8-a015-aac03b32ffd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:18.558438 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.558409 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733a96e9-08bd-48d8-a015-aac03b32ffd6-kube-api-access-6vdcn" pod "733a96e9-08bd-48d8-a015-aac03b32ffd6" (UID: "733a96e9-08bd-48d8-a015-aac03b32ffd6"). InnerVolumeSpecName "kube-api-access-6vdcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:18.587130 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.587103 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:48:18.657176 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.657148 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733a96e9-08bd-48d8-a015-aac03b32ffd6-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.657176 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.657176 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdcn\" (UniqueName: \"kubernetes.io/projected/733a96e9-08bd-48d8-a015-aac03b32ffd6-kube-api-access-6vdcn\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.757953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.757872 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-home\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-home\") pod \"59a0b642-a089-430d-8d23-008780082a49\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " Nov 02 20:48:18.758118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.757976 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a0b642-a089-430d-8d23-008780082a49-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a0b642-a089-430d-8d23-008780082a49-tls-certs\") pod \"59a0b642-a089-430d-8d23-008780082a49\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " Nov 02 20:48:18.758207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.758181 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-home" pod "59a0b642-a089-430d-8d23-008780082a49" (UID: "59a0b642-a089-430d-8d23-008780082a49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:18.758266 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.758235 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-dshm\") pod \"59a0b642-a089-430d-8d23-008780082a49\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " Nov 02 20:48:18.758372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.758355 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a0b642-a089-430d-8d23-008780082a49-kube-api-access-k5sj9\" (UniqueName: \"kubernetes.io/projected/59a0b642-a089-430d-8d23-008780082a49-kube-api-access-k5sj9\") pod \"59a0b642-a089-430d-8d23-008780082a49\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " Nov 02 20:48:18.758426 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.758400 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-model-cache\") pod \"59a0b642-a089-430d-8d23-008780082a49\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " Nov 02 20:48:18.758481 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.758430 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-tmp-dir\") pod \"59a0b642-a089-430d-8d23-008780082a49\" (UID: \"59a0b642-a089-430d-8d23-008780082a49\") " Nov 02 20:48:18.758914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.758842 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-tmp-dir" pod "59a0b642-a089-430d-8d23-008780082a49" (UID: "59a0b642-a089-430d-8d23-008780082a49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:18.759071 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.759010 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.759071 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.759030 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.759245 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.759220 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-model-cache" pod "59a0b642-a089-430d-8d23-008780082a49" (UID: "59a0b642-a089-430d-8d23-008780082a49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:18.760979 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.760954 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a0b642-a089-430d-8d23-008780082a49-tls-certs" pod "59a0b642-a089-430d-8d23-008780082a49" (UID: "59a0b642-a089-430d-8d23-008780082a49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:18.761183 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.761152 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-dshm" pod "59a0b642-a089-430d-8d23-008780082a49" (UID: "59a0b642-a089-430d-8d23-008780082a49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:18.761274 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.761245 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a0b642-a089-430d-8d23-008780082a49-kube-api-access-k5sj9" pod "59a0b642-a089-430d-8d23-008780082a49" (UID: "59a0b642-a089-430d-8d23-008780082a49"). InnerVolumeSpecName "kube-api-access-k5sj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:18.860789 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.860757 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5sj9\" (UniqueName: \"kubernetes.io/projected/59a0b642-a089-430d-8d23-008780082a49-kube-api-access-k5sj9\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.860789 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.860788 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.860986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.860803 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a0b642-a089-430d-8d23-008780082a49-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:18.860986 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:18.860817 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59a0b642-a089-430d-8d23-008780082a49-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:19.368607 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.368578 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" Nov 02 20:48:19.369147 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.368577 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7" event={"ID":"733a96e9-08bd-48d8-a015-aac03b32ffd6","Type":"ContainerDied","Data":"194850f85c185d14d00f540035921c431b2fe4d86bc4c977ecca6a11c784b8a1"} Nov 02 20:48:19.369147 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.368741 5420 scope.go:122] "RemoveContainer" containerID="d3c43470cda3b4879063950db952d63d6123a8e5a0f3c5bfe27f7c8ac03c4fb3" Nov 02 20:48:19.370386 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.370322 5420 generic.go:356] "Generic (PLEG): container finished" podID="59a0b642-a089-430d-8d23-008780082a49" containerID="8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0" exitCode=0 Nov 02 20:48:19.370468 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.370392 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" Nov 02 20:48:19.370468 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.370427 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" event={"ID":"59a0b642-a089-430d-8d23-008780082a49","Type":"ContainerDied","Data":"8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0"} Nov 02 20:48:19.370570 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.370473 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm" event={"ID":"59a0b642-a089-430d-8d23-008780082a49","Type":"ContainerDied","Data":"f118fef896c3e55147204d0aafeb83272a27303b2572a2e13602af59e3fb5a67"} Nov 02 20:48:19.379967 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.379947 5420 scope.go:122] "RemoveContainer" containerID="8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0" Nov 02 20:48:19.392953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.392928 5420 scope.go:122] "RemoveContainer" containerID="8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0" Nov 02 20:48:19.393509 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:19.393480 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0\": container with ID starting with 8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0 not found: ID does not exist" containerID="8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0" Nov 02 20:48:19.393602 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.393517 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0"} err="failed to get container status \"8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0\": rpc error: code = NotFound desc = could not find container \"8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0\": container with ID starting with 8d9065d1fe05808ade511a498743580ee8d99c7019150aadf4a03f565e75ddd0 not found: ID does not exist" Nov 02 20:48:19.521719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.521646 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm"] Nov 02 20:48:19.568316 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.568289 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2250b4/distinct-w1-2250b4-kserve-775ccc87d7-dg5fm"] Nov 02 20:48:19.671281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.671251 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7"] Nov 02 20:48:19.675580 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.675555 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-30ee89-9b8f6b86b-8hll7"] Nov 02 20:48:19.776665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.776594 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a0b642-a089-430d-8d23-008780082a49" path="/var/lib/kubelet/pods/59a0b642-a089-430d-8d23-008780082a49/volumes" Nov 02 20:48:19.777027 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:19.777012 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733a96e9-08bd-48d8-a015-aac03b32ffd6" path="/var/lib/kubelet/pods/733a96e9-08bd-48d8-a015-aac03b32ffd6/volumes" Nov 02 20:48:25.714082 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:25.714049 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v"] Nov 02 20:48:25.714619 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:25.714242 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" containerID="cri-o://31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b" gracePeriod=30 Nov 02 20:48:26.517966 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:26.517935 5420 patch_prober.go:29] interesting pod/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:48:26.518127 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:26.517985 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:48:30.975528 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:30.975501 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:48:31.109856 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.109827 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-data\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.109991 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.109868 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a52f9ae8-24ab-4006-b444-9e3971071196-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52f9ae8-24ab-4006-b444-9e3971071196-istiod-ca-cert\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.109991 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.109912 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-credential-socket\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.109991 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.109937 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-kube-api-access-xfbmv\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-kube-api-access-xfbmv\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.110104 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110036 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-istio-token\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-istio-token\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.110104 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110085 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-socket\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.110104 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110085 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-data" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:31.110271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110119 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-envoy\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.110271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110155 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a52f9ae8-24ab-4006-b444-9e3971071196-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52f9ae8-24ab-4006-b444-9e3971071196-istio-podinfo\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.110271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110202 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-certs\") pod \"a52f9ae8-24ab-4006-b444-9e3971071196\" (UID: \"a52f9ae8-24ab-4006-b444-9e3971071196\") " Nov 02 20:48:31.110271 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110222 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-credential-socket" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:31.110478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110321 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a52f9ae8-24ab-4006-b444-9e3971071196-istiod-ca-cert" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:31.110478 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110404 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-socket" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:31.110660 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110637 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-certs" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:31.110799 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110655 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a52f9ae8-24ab-4006-b444-9e3971071196-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.110799 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110721 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.110799 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110737 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.110799 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.110751 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.112195 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.112175 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a52f9ae8-24ab-4006-b444-9e3971071196-istio-podinfo" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:48:31.112329 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.112312 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-istio-token" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:31.112621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.112606 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-envoy" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:31.112683 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.112618 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-kube-api-access-xfbmv" pod "a52f9ae8-24ab-4006-b444-9e3971071196" (UID: "a52f9ae8-24ab-4006-b444-9e3971071196"). InnerVolumeSpecName "kube-api-access-xfbmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:31.211859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.211825 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.211859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.211852 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.211859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.211865 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a52f9ae8-24ab-4006-b444-9e3971071196-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.212091 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.211877 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a52f9ae8-24ab-4006-b444-9e3971071196-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.212091 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.211890 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfbmv\" (UniqueName: \"kubernetes.io/projected/a52f9ae8-24ab-4006-b444-9e3971071196-kube-api-access-xfbmv\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:31.460871 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.460787 5420 generic.go:356] "Generic (PLEG): container finished" podID="a52f9ae8-24ab-4006-b444-9e3971071196" containerID="31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b" exitCode=0 Nov 02 20:48:31.461019 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.460888 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" Nov 02 20:48:31.461108 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.460958 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" event={"ID":"a52f9ae8-24ab-4006-b444-9e3971071196","Type":"ContainerDied","Data":"31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b"} Nov 02 20:48:31.461166 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.461157 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v" event={"ID":"a52f9ae8-24ab-4006-b444-9e3971071196","Type":"ContainerDied","Data":"22ebb6736948c601ad460776da09da9cae0848f8b4191b822a449a4a72bf5ded"} Nov 02 20:48:31.461221 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.461181 5420 scope.go:122] "RemoveContainer" containerID="31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b" Nov 02 20:48:31.474145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.474129 5420 scope.go:122] "RemoveContainer" containerID="31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b" Nov 02 20:48:31.474594 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:31.474572 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b\": container with ID starting with 31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b not found: ID does not exist" containerID="31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b" Nov 02 20:48:31.474762 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.474735 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b"} err="failed to get container status \"31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b\": rpc error: code = NotFound desc = could not find container \"31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b\": container with ID starting with 31f864b049374a25303bc0bcb43dc76bb1cc327c1845e2e649b7d839806d1e6b not found: ID does not exist" Nov 02 20:48:31.615276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.615243 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v"] Nov 02 20:48:31.618998 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.618970 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a5cad4-openshift-default-7d6c4cd4fc-jjm4v"] Nov 02 20:48:31.776153 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:31.776078 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" path="/var/lib/kubelet/pods/a52f9ae8-24ab-4006-b444-9e3971071196/volumes" Nov 02 20:48:32.239033 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.239005 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr"] Nov 02 20:48:32.239482 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.239225 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" podUID="9e453b0a-c58d-4733-93ec-935015f97d41" containerName="maas-api" containerID="cri-o://7198163d617508ede8d4354783dc9cc2a16a0aa715b2194627a2aa077b61f742" gracePeriod=30 Nov 02 20:48:32.481417 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.481383 5420 generic.go:356] "Generic (PLEG): container finished" podID="9e453b0a-c58d-4733-93ec-935015f97d41" containerID="7198163d617508ede8d4354783dc9cc2a16a0aa715b2194627a2aa077b61f742" exitCode=0 Nov 02 20:48:32.481608 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.481552 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" event={"ID":"9e453b0a-c58d-4733-93ec-935015f97d41","Type":"ContainerDied","Data":"7198163d617508ede8d4354783dc9cc2a16a0aa715b2194627a2aa077b61f742"} Nov 02 20:48:32.535460 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.535436 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:48:32.634315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.630116 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e453b0a-c58d-4733-93ec-935015f97d41-kube-api-access-29tt9\" (UniqueName: \"kubernetes.io/projected/9e453b0a-c58d-4733-93ec-935015f97d41-kube-api-access-29tt9\") pod \"9e453b0a-c58d-4733-93ec-935015f97d41\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " Nov 02 20:48:32.634315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.630227 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e453b0a-c58d-4733-93ec-935015f97d41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e453b0a-c58d-4733-93ec-935015f97d41-openshift-service-ca\") pod \"9e453b0a-c58d-4733-93ec-935015f97d41\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " Nov 02 20:48:32.634315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.630312 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls\") pod \"9e453b0a-c58d-4733-93ec-935015f97d41\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " Nov 02 20:48:32.634315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.630397 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-metrics-certs\") pod \"9e453b0a-c58d-4733-93ec-935015f97d41\" (UID: \"9e453b0a-c58d-4733-93ec-935015f97d41\") " Nov 02 20:48:32.634315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.631453 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e453b0a-c58d-4733-93ec-935015f97d41-openshift-service-ca" pod "9e453b0a-c58d-4733-93ec-935015f97d41" (UID: "9e453b0a-c58d-4733-93ec-935015f97d41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:32.648901 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.648852 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e453b0a-c58d-4733-93ec-935015f97d41-kube-api-access-29tt9" pod "9e453b0a-c58d-4733-93ec-935015f97d41" (UID: "9e453b0a-c58d-4733-93ec-935015f97d41"). InnerVolumeSpecName "kube-api-access-29tt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:32.661043 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.659824 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls" pod "9e453b0a-c58d-4733-93ec-935015f97d41" (UID: "9e453b0a-c58d-4733-93ec-935015f97d41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:32.661043 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.659912 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-metrics-certs" pod "9e453b0a-c58d-4733-93ec-935015f97d41" (UID: "9e453b0a-c58d-4733-93ec-935015f97d41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:32.697316 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:32.697280 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:48:32.732299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.732268 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:32.732299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.732291 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e453b0a-c58d-4733-93ec-935015f97d41-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:32.732299 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.732301 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29tt9\" (UniqueName: \"kubernetes.io/projected/9e453b0a-c58d-4733-93ec-935015f97d41-kube-api-access-29tt9\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:32.732492 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:32.732312 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e453b0a-c58d-4733-93ec-935015f97d41-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:33.490445 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:33.490417 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" Nov 02 20:48:33.490873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:33.490412 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr" event={"ID":"9e453b0a-c58d-4733-93ec-935015f97d41","Type":"ContainerDied","Data":"041c3a6c47c05171bd6ba7027d0b83ba355302606dd9207cb8c09860b9865c08"} Nov 02 20:48:33.490873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:33.490546 5420 scope.go:122] "RemoveContainer" containerID="7198163d617508ede8d4354783dc9cc2a16a0aa715b2194627a2aa077b61f742" Nov 02 20:48:33.643709 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:33.643654 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr"] Nov 02 20:48:33.689981 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:33.689957 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-4166b0-779c8c6fd-tsnbr"] Nov 02 20:48:33.775532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:33.775472 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e453b0a-c58d-4733-93ec-935015f97d41" path="/var/lib/kubelet/pods/9e453b0a-c58d-4733-93ec-935015f97d41/volumes" Nov 02 20:48:42.224886 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:42.224808 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659"] Nov 02 20:48:42.225302 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:42.225011 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" containerID="cri-o://3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e" gracePeriod=30 Nov 02 20:48:47.513575 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.513545 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:48:47.584989 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.584959 5420 generic.go:356] "Generic (PLEG): container finished" podID="37050465-6348-48cb-90bd-36272f806434" containerID="3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e" exitCode=0 Nov 02 20:48:47.585149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.585000 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" event={"ID":"37050465-6348-48cb-90bd-36272f806434","Type":"ContainerDied","Data":"3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e"} Nov 02 20:48:47.585149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.585034 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" Nov 02 20:48:47.585149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.585082 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659" event={"ID":"37050465-6348-48cb-90bd-36272f806434","Type":"ContainerDied","Data":"81fd5eb35d593acb09a0e9e2f05ad43da8f5fa4d4631a8633f43ee6246cc5d49"} Nov 02 20:48:47.585149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.585116 5420 scope.go:122] "RemoveContainer" containerID="3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e" Nov 02 20:48:47.591372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591350 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-credential-socket\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591383 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-certs\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591409 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-envoy\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591442 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-istio-token\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-istio-token\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591469 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-kube-api-access-cxw69\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-kube-api-access-cxw69\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591614 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37050465-6348-48cb-90bd-36272f806434-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37050465-6348-48cb-90bd-36272f806434-istio-podinfo\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591681 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-data\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591662 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-credential-socket" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:47.591876 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591761 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37050465-6348-48cb-90bd-36272f806434-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37050465-6348-48cb-90bd-36272f806434-istiod-ca-cert\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591876 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591785 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-socket\") pod \"37050465-6348-48cb-90bd-36272f806434\" (UID: \"37050465-6348-48cb-90bd-36272f806434\") " Nov 02 20:48:47.591876 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.591824 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-certs" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:47.592072 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.592013 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-data" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:47.592200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.592180 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.592200 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.592191 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-socket" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:47.592315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.592205 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.592315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.592221 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.592315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.592259 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37050465-6348-48cb-90bd-36272f806434-istiod-ca-cert" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:47.594094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.594065 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-kube-api-access-cxw69" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "kube-api-access-cxw69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:47.594424 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.594404 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-envoy" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:48:47.594823 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.594794 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-istio-token" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:47.595148 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.595119 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37050465-6348-48cb-90bd-36272f806434-istio-podinfo" pod "37050465-6348-48cb-90bd-36272f806434" (UID: "37050465-6348-48cb-90bd-36272f806434"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:48:47.596162 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.596144 5420 scope.go:122] "RemoveContainer" containerID="e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d" Nov 02 20:48:47.616575 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.616556 5420 scope.go:122] "RemoveContainer" containerID="3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e" Nov 02 20:48:47.616892 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:47.616867 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e\": container with ID starting with 3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e not found: ID does not exist" containerID="3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e" Nov 02 20:48:47.616967 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.616894 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e"} err="failed to get container status \"3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e\": rpc error: code = NotFound desc = could not find container \"3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e\": container with ID starting with 3a793249b4705118ba60953e7d5b303a603069274dcdad21e532b10bc1db643e not found: ID does not exist" Nov 02 20:48:47.616967 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.616910 5420 scope.go:122] "RemoveContainer" containerID="e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d" Nov 02 20:48:47.617217 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:47.617193 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d\": container with ID starting with e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d not found: ID does not exist" containerID="e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d" Nov 02 20:48:47.617304 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.617222 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d"} err="failed to get container status \"e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d\": rpc error: code = NotFound desc = could not find container \"e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d\": container with ID starting with e09eed8abead2793935a815578a8b5d975f86bc7480b7606fc5e158d82ac6f6d not found: ID does not exist" Nov 02 20:48:47.693414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.693383 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.693414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.693416 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.693665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.693433 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxw69\" (UniqueName: \"kubernetes.io/projected/37050465-6348-48cb-90bd-36272f806434-kube-api-access-cxw69\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.693665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.693448 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37050465-6348-48cb-90bd-36272f806434-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.693665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.693461 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37050465-6348-48cb-90bd-36272f806434-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.693665 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.693474 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37050465-6348-48cb-90bd-36272f806434-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:47.992527 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.992497 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659"] Nov 02 20:48:47.996381 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:47.996355 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-4166b0-openshift-default-78b7f9f559-cm659"] Nov 02 20:48:49.774590 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:49.774563 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37050465-6348-48cb-90bd-36272f806434" path="/var/lib/kubelet/pods/37050465-6348-48cb-90bd-36272f806434/volumes" Nov 02 20:48:50.710542 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:50.710507 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h"] Nov 02 20:48:50.710800 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:50.710752 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" podUID="bf33d81f-c923-417e-a5d7-48217518b55c" containerName="maas-api" containerID="cri-o://edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e" gracePeriod=30 Nov 02 20:48:51.016799 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.016776 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:48:51.130914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.130881 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf33d81f-c923-417e-a5d7-48217518b55c-kube-api-access-jl2s2\" (UniqueName: \"kubernetes.io/projected/bf33d81f-c923-417e-a5d7-48217518b55c-kube-api-access-jl2s2\") pod \"bf33d81f-c923-417e-a5d7-48217518b55c\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " Nov 02 20:48:51.131123 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.131029 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls\") pod \"bf33d81f-c923-417e-a5d7-48217518b55c\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " Nov 02 20:48:51.131123 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.131107 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-metrics-certs\") pod \"bf33d81f-c923-417e-a5d7-48217518b55c\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " Nov 02 20:48:51.131247 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.131133 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf33d81f-c923-417e-a5d7-48217518b55c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf33d81f-c923-417e-a5d7-48217518b55c-openshift-service-ca\") pod \"bf33d81f-c923-417e-a5d7-48217518b55c\" (UID: \"bf33d81f-c923-417e-a5d7-48217518b55c\") " Nov 02 20:48:51.131735 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.131703 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf33d81f-c923-417e-a5d7-48217518b55c-openshift-service-ca" pod "bf33d81f-c923-417e-a5d7-48217518b55c" (UID: "bf33d81f-c923-417e-a5d7-48217518b55c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:51.133699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.133644 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls" pod "bf33d81f-c923-417e-a5d7-48217518b55c" (UID: "bf33d81f-c923-417e-a5d7-48217518b55c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:51.134845 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.134818 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf33d81f-c923-417e-a5d7-48217518b55c-kube-api-access-jl2s2" pod "bf33d81f-c923-417e-a5d7-48217518b55c" (UID: "bf33d81f-c923-417e-a5d7-48217518b55c"). InnerVolumeSpecName "kube-api-access-jl2s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:51.135094 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.135069 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-metrics-certs" pod "bf33d81f-c923-417e-a5d7-48217518b55c" (UID: "bf33d81f-c923-417e-a5d7-48217518b55c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:51.233139 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.233118 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:51.233139 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.233139 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf33d81f-c923-417e-a5d7-48217518b55c-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:51.233279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.233151 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2s2\" (UniqueName: \"kubernetes.io/projected/bf33d81f-c923-417e-a5d7-48217518b55c-kube-api-access-jl2s2\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:51.233279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.233159 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf33d81f-c923-417e-a5d7-48217518b55c-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:51.611579 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.611544 5420 generic.go:356] "Generic (PLEG): container finished" podID="bf33d81f-c923-417e-a5d7-48217518b55c" containerID="edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e" exitCode=0 Nov 02 20:48:51.611742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.611593 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" Nov 02 20:48:51.611742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.611627 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" event={"ID":"bf33d81f-c923-417e-a5d7-48217518b55c","Type":"ContainerDied","Data":"edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e"} Nov 02 20:48:51.611742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.611694 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h" event={"ID":"bf33d81f-c923-417e-a5d7-48217518b55c","Type":"ContainerDied","Data":"8cba87ebb5b816022ec56b22e3d9de9cdbefde2bb2c03d54c4b4f65727d222b5"} Nov 02 20:48:51.611742 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.611719 5420 scope.go:122] "RemoveContainer" containerID="edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e" Nov 02 20:48:51.622604 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.622585 5420 scope.go:122] "RemoveContainer" containerID="edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e" Nov 02 20:48:51.622900 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:51.622881 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e\": container with ID starting with edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e not found: ID does not exist" containerID="edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e" Nov 02 20:48:51.622959 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.622909 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e"} err="failed to get container status \"edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e\": rpc error: code = NotFound desc = could not find container \"edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e\": container with ID starting with edaaca80fef6f97ba5a107d0c0d4d7f444d7b139f543ca6b364233cec80c640e not found: ID does not exist" Nov 02 20:48:51.767036 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.766993 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h"] Nov 02 20:48:51.811101 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:51.811068 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-30ee89-7bbdfbd875-r7r4h"] Nov 02 20:48:53.776149 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:53.776117 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf33d81f-c923-417e-a5d7-48217518b55c" path="/var/lib/kubelet/pods/bf33d81f-c923-417e-a5d7-48217518b55c/volumes" Nov 02 20:48:58.364059 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.364027 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc"] Nov 02 20:48:58.364532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.364202 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" podUID="b976fa43-0c79-4006-8162-d63a571e96b0" containerName="maas-api" containerID="cri-o://68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38" gracePeriod=30 Nov 02 20:48:58.550343 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.550313 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct"] Nov 02 20:48:58.566011 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.565979 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn"] Nov 02 20:48:58.636558 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.636535 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:48:58.666582 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.666555 5420 generic.go:356] "Generic (PLEG): container finished" podID="b976fa43-0c79-4006-8162-d63a571e96b0" containerID="68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38" exitCode=0 Nov 02 20:48:58.666729 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.666597 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" Nov 02 20:48:58.666729 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.666632 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" event={"ID":"b976fa43-0c79-4006-8162-d63a571e96b0","Type":"ContainerDied","Data":"68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38"} Nov 02 20:48:58.666729 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.666709 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc" event={"ID":"b976fa43-0c79-4006-8162-d63a571e96b0","Type":"ContainerDied","Data":"57b211a2d8f3d54c118028adc59375133b4ab4d3577c7a5d5ae23a5bb0ac0535"} Nov 02 20:48:58.666858 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.666730 5420 scope.go:122] "RemoveContainer" containerID="68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38" Nov 02 20:48:58.675956 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.675942 5420 scope.go:122] "RemoveContainer" containerID="68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38" Nov 02 20:48:58.676227 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:48:58.676208 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38\": container with ID starting with 68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38 not found: ID does not exist" containerID="68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38" Nov 02 20:48:58.676277 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.676236 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38"} err="failed to get container status \"68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38\": rpc error: code = NotFound desc = could not find container \"68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38\": container with ID starting with 68e2476d2bcff904fef1c021433e3bcc29b405358b4478a686df144a98207e38 not found: ID does not exist" Nov 02 20:48:58.728217 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.728189 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-metrics-certs\") pod \"b976fa43-0c79-4006-8162-d63a571e96b0\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " Nov 02 20:48:58.728403 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.728270 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b976fa43-0c79-4006-8162-d63a571e96b0-kube-api-access-pv4n4\" (UniqueName: \"kubernetes.io/projected/b976fa43-0c79-4006-8162-d63a571e96b0-kube-api-access-pv4n4\") pod \"b976fa43-0c79-4006-8162-d63a571e96b0\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " Nov 02 20:48:58.728403 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.728316 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b976fa43-0c79-4006-8162-d63a571e96b0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b976fa43-0c79-4006-8162-d63a571e96b0-openshift-service-ca\") pod \"b976fa43-0c79-4006-8162-d63a571e96b0\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " Nov 02 20:48:58.728403 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.728384 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls\") pod \"b976fa43-0c79-4006-8162-d63a571e96b0\" (UID: \"b976fa43-0c79-4006-8162-d63a571e96b0\") " Nov 02 20:48:58.728819 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.728788 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b976fa43-0c79-4006-8162-d63a571e96b0-openshift-service-ca" pod "b976fa43-0c79-4006-8162-d63a571e96b0" (UID: "b976fa43-0c79-4006-8162-d63a571e96b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:48:58.730551 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.730522 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-metrics-certs" pod "b976fa43-0c79-4006-8162-d63a571e96b0" (UID: "b976fa43-0c79-4006-8162-d63a571e96b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:58.730551 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.730529 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b976fa43-0c79-4006-8162-d63a571e96b0-kube-api-access-pv4n4" pod "b976fa43-0c79-4006-8162-d63a571e96b0" (UID: "b976fa43-0c79-4006-8162-d63a571e96b0"). InnerVolumeSpecName "kube-api-access-pv4n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:48:58.730773 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.730594 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls" pod "b976fa43-0c79-4006-8162-d63a571e96b0" (UID: "b976fa43-0c79-4006-8162-d63a571e96b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:48:58.829800 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.829708 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.829800 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.829745 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b976fa43-0c79-4006-8162-d63a571e96b0-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.829800 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.829758 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv4n4\" (UniqueName: \"kubernetes.io/projected/b976fa43-0c79-4006-8162-d63a571e96b0-kube-api-access-pv4n4\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:58.829800 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:58.829772 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b976fa43-0c79-4006-8162-d63a571e96b0-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:48:59.079448 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:59.079417 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc"] Nov 02 20:48:59.083600 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:59.083573 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-58c2e0-77d598fc8c-sqdfc"] Nov 02 20:48:59.779256 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:48:59.779221 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b976fa43-0c79-4006-8162-d63a571e96b0" path="/var/lib/kubelet/pods/b976fa43-0c79-4006-8162-d63a571e96b0/volumes" Nov 02 20:49:00.528894 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:00.528861 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf"] Nov 02 20:49:00.529078 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:00.529058 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" containerID="cri-o://1469422394fa16f8cfffc41651dd99ffe158f8b9b749f955a0c1fd53c9d442cc" gracePeriod=30 Nov 02 20:49:00.537325 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:00.537300 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz"] Nov 02 20:49:00.537483 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:00.537466 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" containerID="cri-o://efc9df80052f5a624e93d7f54b8abee6aaa4bee100c4bec2cb56875ba395c889" gracePeriod=30 Nov 02 20:49:01.389199 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:01.389162 5420 patch_prober.go:29] interesting pod/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:49:01.389599 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:01.389234 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:49:02.658037 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:02.657997 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:49:05.714283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.713951 5420 generic.go:356] "Generic (PLEG): container finished" podID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerID="1469422394fa16f8cfffc41651dd99ffe158f8b9b749f955a0c1fd53c9d442cc" exitCode=0 Nov 02 20:49:05.714283 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.714197 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" event={"ID":"a9bddc36-c7cd-4413-8b70-d45e207a399a","Type":"ContainerDied","Data":"1469422394fa16f8cfffc41651dd99ffe158f8b9b749f955a0c1fd53c9d442cc"} Nov 02 20:49:05.717272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.717078 5420 generic.go:356] "Generic (PLEG): container finished" podID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerID="efc9df80052f5a624e93d7f54b8abee6aaa4bee100c4bec2cb56875ba395c889" exitCode=0 Nov 02 20:49:05.717272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.717133 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" event={"ID":"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3","Type":"ContainerDied","Data":"efc9df80052f5a624e93d7f54b8abee6aaa4bee100c4bec2cb56875ba395c889"} Nov 02 20:49:05.717272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.717179 5420 scope.go:122] "RemoveContainer" containerID="892f37a058392d43b648e78c0ed91a358c46eb9ad26bca11132f5990a2fb029e" Nov 02 20:49:05.823079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.823050 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:49:05.836182 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.836163 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:49:05.910142 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910116 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-certs\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910169 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-credential-socket\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910210 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-token\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-token\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910281 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910246 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9bddc36-c7cd-4413-8b70-d45e207a399a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bddc36-c7cd-4413-8b70-d45e207a399a-istiod-ca-cert\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910291 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-podinfo\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910313 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-envoy\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910336 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-socket\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910366 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-certs" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.910456 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910379 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-data\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910737 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910517 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-kube-api-access-pzhv6\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-kube-api-access-pzhv6\") pod \"a9bddc36-c7cd-4413-8b70-d45e207a399a\" (UID: \"a9bddc36-c7cd-4413-8b70-d45e207a399a\") " Nov 02 20:49:05.910737 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910529 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-credential-socket" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.910839 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.910772 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-socket" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.912034 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.912004 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.912146 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.912038 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.912146 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.912057 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:05.912146 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.912087 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9bddc36-c7cd-4413-8b70-d45e207a399a-istiod-ca-cert" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:05.912146 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.912104 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-data" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.912780 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.912655 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-podinfo" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:49:05.912886 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.912807 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-envoy" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:05.912947 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.912912 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-token" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:05.921335 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:05.921308 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-kube-api-access-pzhv6" pod "a9bddc36-c7cd-4413-8b70-d45e207a399a" (UID: "a9bddc36-c7cd-4413-8b70-d45e207a399a"). InnerVolumeSpecName "kube-api-access-pzhv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:06.012498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.012465 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-kube-api-access-xqwdm\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-kube-api-access-xqwdm\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.012633 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.012552 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-socket\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.012633 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.012590 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istiod-ca-cert\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.012779 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.012657 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-credential-socket\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.012779 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.012715 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-token\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-token\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.012779 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.012771 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-data\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.012941 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.012804 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-podinfo\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.012941 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.012842 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-socket" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:06.013137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013117 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-data" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:06.013220 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013200 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-credential-socket" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:06.013278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013232 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istiod-ca-cert" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:06.013278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013253 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-certs\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.013397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013358 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-envoy\") pod \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\" (UID: \"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3\") " Nov 02 20:49:06.013449 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013436 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-certs" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:06.013795 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013775 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.013893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013798 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9bddc36-c7cd-4413-8b70-d45e207a399a-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.013893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013813 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.013893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013829 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.013893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013841 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9bddc36-c7cd-4413-8b70-d45e207a399a-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.013893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013854 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.013893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013867 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.013893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013879 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhv6\" (UniqueName: \"kubernetes.io/projected/a9bddc36-c7cd-4413-8b70-d45e207a399a-kube-api-access-pzhv6\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.013893 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013893 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.014315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013907 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.014315 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.013920 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.015485 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.015464 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-envoy" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:06.015591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.015513 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-kube-api-access-xqwdm" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "kube-api-access-xqwdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:06.015744 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.015726 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-podinfo" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:49:06.015998 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.015979 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-token" pod "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" (UID: "4c61ef80-a2e5-4319-8c6f-ba79be60bdf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:06.115500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.115470 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.115500 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.115499 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwdm\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-kube-api-access-xqwdm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.115711 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.115510 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.115711 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.115519 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:06.727900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.727855 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" Nov 02 20:49:06.727900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.727861 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz" event={"ID":"4c61ef80-a2e5-4319-8c6f-ba79be60bdf3","Type":"ContainerDied","Data":"8a54fd5da083a5c77265b0a7716665d6867eb4498cda4b783e09e2cb1728609d"} Nov 02 20:49:06.727900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.728052 5420 scope.go:122] "RemoveContainer" containerID="efc9df80052f5a624e93d7f54b8abee6aaa4bee100c4bec2cb56875ba395c889" Nov 02 20:49:06.731006 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.730976 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" event={"ID":"a9bddc36-c7cd-4413-8b70-d45e207a399a","Type":"ContainerDied","Data":"613b35c5fb66d49b03e8a541396d93c235032574f6d1fe2b5c52b42a8c9d1818"} Nov 02 20:49:06.733708 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.732531 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf" Nov 02 20:49:06.741732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.741715 5420 scope.go:122] "RemoveContainer" containerID="1469422394fa16f8cfffc41651dd99ffe158f8b9b749f955a0c1fd53c9d442cc" Nov 02 20:49:06.889725 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.889653 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz"] Nov 02 20:49:06.936831 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.936801 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-pfsrz"] Nov 02 20:49:06.982634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.963629 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n"] Nov 02 20:49:06.982634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.963861 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" containerID="cri-o://ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9" gracePeriod=30 Nov 02 20:49:06.982634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.982013 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb"] Nov 02 20:49:06.982634 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:06.982272 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" containerID="cri-o://792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a" gracePeriod=30 Nov 02 20:49:07.000459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:07.000430 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf"] Nov 02 20:49:07.007816 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:07.007791 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-30ee89-openshift-default-57474bd7b9-qdzpf"] Nov 02 20:49:07.781691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:07.778636 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" path="/var/lib/kubelet/pods/4c61ef80-a2e5-4319-8c6f-ba79be60bdf3/volumes" Nov 02 20:49:07.781691 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:07.779308 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" path="/var/lib/kubelet/pods/a9bddc36-c7cd-4413-8b70-d45e207a399a/volumes" Nov 02 20:49:10.134947 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.134918 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc"] Nov 02 20:49:10.135378 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135364 5420 cpu_manager.go:414] "containerMap: removing container" podUID="3595205a-4595-486b-996d-38dbda6359c1" containerName="maas-api" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135381 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3595205a-4595-486b-996d-38dbda6359c1" containerName="maas-api" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135395 5420 cpu_manager.go:414] "containerMap: removing container" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135400 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135408 5420 cpu_manager.go:414] "containerMap: removing container" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135412 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135422 5420 cpu_manager.go:414] "containerMap: removing container" podUID="733a96e9-08bd-48d8-a015-aac03b32ffd6" containerName="payload-processing" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135427 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733a96e9-08bd-48d8-a015-aac03b32ffd6" containerName="payload-processing" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135434 5420 cpu_manager.go:414] "containerMap: removing container" podUID="9e453b0a-c58d-4733-93ec-935015f97d41" containerName="maas-api" Nov 02 20:49:10.135444 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135438 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e453b0a-c58d-4733-93ec-935015f97d41" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135449 5420 cpu_manager.go:414] "containerMap: removing container" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135456 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135466 5420 cpu_manager.go:414] "containerMap: removing container" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135470 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135482 5420 cpu_manager.go:414] "containerMap: removing container" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135486 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135492 5420 cpu_manager.go:414] "containerMap: removing container" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135497 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135503 5420 cpu_manager.go:414] "containerMap: removing container" podUID="0afab02f-589b-473a-91d8-35ee552bb40a" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135507 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0afab02f-589b-473a-91d8-35ee552bb40a" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135513 5420 cpu_manager.go:414] "containerMap: removing container" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135517 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135523 5420 cpu_manager.go:414] "containerMap: removing container" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135528 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135541 5420 cpu_manager.go:414] "containerMap: removing container" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135545 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135552 5420 cpu_manager.go:414] "containerMap: removing container" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135555 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135562 5420 cpu_manager.go:414] "containerMap: removing container" podUID="2e38931c-fe75-457c-b808-03eb5dbb89d8" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135566 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e38931c-fe75-457c-b808-03eb5dbb89d8" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135571 5420 cpu_manager.go:414] "containerMap: removing container" podUID="b976fa43-0c79-4006-8162-d63a571e96b0" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135576 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b976fa43-0c79-4006-8162-d63a571e96b0" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135585 5420 cpu_manager.go:414] "containerMap: removing container" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135589 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135596 5420 cpu_manager.go:414] "containerMap: removing container" podUID="90ab3a61-5a80-4946-841b-c937011affc4" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135600 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ab3a61-5a80-4946-841b-c937011affc4" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135611 5420 cpu_manager.go:414] "containerMap: removing container" podUID="3bf67e9e-2c20-42bd-869d-e4637795eb21" containerName="payload-processing" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135618 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf67e9e-2c20-42bd-869d-e4637795eb21" containerName="payload-processing" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135626 5420 cpu_manager.go:414] "containerMap: removing container" podUID="bf33d81f-c923-417e-a5d7-48217518b55c" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135631 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf33d81f-c923-417e-a5d7-48217518b55c" containerName="maas-api" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135637 5420 cpu_manager.go:414] "containerMap: removing container" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135641 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135650 5420 cpu_manager.go:414] "containerMap: removing container" podUID="ef16c845-34db-4717-bae8-173ec711aa55" containerName="payload-pre-processing" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135654 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef16c845-34db-4717-bae8-173ec711aa55" containerName="payload-pre-processing" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135664 5420 cpu_manager.go:414] "containerMap: removing container" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135679 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135688 5420 cpu_manager.go:414] "containerMap: removing container" podUID="f4761389-0a5c-4fbf-a988-3a5c7e45c80d" containerName="payload-processing" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135695 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4761389-0a5c-4fbf-a988-3a5c7e45c80d" containerName="payload-processing" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135713 5420 cpu_manager.go:414] "containerMap: removing container" podUID="1e74e222-c059-4021-910f-78d9819015c2" containerName="payload-processing" Nov 02 20:49:10.135764 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135718 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e74e222-c059-4021-910f-78d9819015c2" containerName="payload-processing" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135789 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a9bf42-6667-4ed6-b824-b4b4f209f989" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135797 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0afab02f-589b-473a-91d8-35ee552bb40a" containerName="maas-api" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135804 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="063bf910-ea3f-4d1c-ac5b-d862a108a343" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135812 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135818 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52f9ae8-24ab-4006-b444-9e3971071196" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135824 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135828 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e38931c-fe75-457c-b808-03eb5dbb89d8" containerName="maas-api" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135832 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b976fa43-0c79-4006-8162-d63a571e96b0" containerName="maas-api" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135838 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8accdb-1605-45c0-9402-8299d48e0b31" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135872 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ab3a61-5a80-4946-841b-c937011affc4" containerName="maas-api" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135880 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf67e9e-2c20-42bd-869d-e4637795eb21" containerName="payload-processing" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135885 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf33d81f-c923-417e-a5d7-48217518b55c" containerName="maas-api" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135891 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a0b642-a089-430d-8d23-008780082a49" containerName="main" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135897 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef16c845-34db-4717-bae8-173ec711aa55" containerName="payload-pre-processing" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135903 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135908 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4761389-0a5c-4fbf-a988-3a5c7e45c80d" containerName="payload-processing" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135914 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135922 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e74e222-c059-4021-910f-78d9819015c2" containerName="payload-processing" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135933 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3595205a-4595-486b-996d-38dbda6359c1" containerName="maas-api" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135941 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa56fb7-abaa-4169-9c0a-319feee7dc86" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135945 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135951 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="733a96e9-08bd-48d8-a015-aac03b32ffd6" containerName="payload-processing" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135956 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e453b0a-c58d-4733-93ec-935015f97d41" containerName="maas-api" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135963 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="37050465-6348-48cb-90bd-36272f806434" containerName="istio-proxy" Nov 02 20:49:10.137371 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.135969 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9bddc36-c7cd-4413-8b70-d45e207a399a" containerName="istio-proxy" Nov 02 20:49:10.140573 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.140557 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.160219 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.160193 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gzs\" (UniqueName: \"kubernetes.io/projected/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-kube-api-access-x5gzs\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.160347 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.160239 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.160347 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.160322 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.193740 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.193714 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc"] Nov 02 20:49:10.235992 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.235958 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 20:49:10.236153 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.236138 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 20:49:10.261796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.261774 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gzs\" (UniqueName: \"kubernetes.io/projected/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-kube-api-access-x5gzs\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.261890 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.261819 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.261890 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.261841 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.262376 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.262357 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.264048 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.264028 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.318611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.318592 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gzs\" (UniqueName: \"kubernetes.io/projected/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-kube-api-access-x5gzs\") pod \"maas-api-revoke-sub-33d935bb-8n4nc\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.455495 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.455439 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:10.819796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:10.819774 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc"] Nov 02 20:49:10.821610 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:49:10.821582 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca59f5f_cbe8_49a5_8a0a_80ad1b525baf.slice/crio-b756901162f17bcf18909411015cdda1fbff1be18346c4c9e435d200cfb56489 WatchSource:0}: Error finding container b756901162f17bcf18909411015cdda1fbff1be18346c4c9e435d200cfb56489: Status 404 returned error can't find the container with id b756901162f17bcf18909411015cdda1fbff1be18346c4c9e435d200cfb56489 Nov 02 20:49:11.524728 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:11.524697 5420 patch_prober.go:29] interesting pod/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:49:11.525066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:11.524751 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:49:11.769169 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:11.769136 5420 generic.go:356] "Generic (PLEG): container finished" podID="8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf" containerID="e71d083b129cb1cee3e47d21d21bff55f28ef2600b5543601167b78c65d21bd9" exitCode=0 Nov 02 20:49:11.773552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:11.773493 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" event={"ID":"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf","Type":"ContainerDied","Data":"e71d083b129cb1cee3e47d21d21bff55f28ef2600b5543601167b78c65d21bd9"} Nov 02 20:49:11.773552 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:11.773536 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" event={"ID":"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf","Type":"ContainerStarted","Data":"b756901162f17bcf18909411015cdda1fbff1be18346c4c9e435d200cfb56489"} Nov 02 20:49:12.240014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.239993 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:49:12.261983 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.261964 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:49:12.280760 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.280739 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf245911-b48a-4645-94e1-10a64054febd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf245911-b48a-4645-94e1-10a64054febd-istio-podinfo\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.280869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.280771 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-kube-api-access-589q2\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-kube-api-access-589q2\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.280869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.280801 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-data\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.280869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.280830 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-token\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-token\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.281039 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.280872 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-certs\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.281039 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.280913 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-socket\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.281039 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.280941 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-certs\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.281039 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.280981 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istiod-ca-cert\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281072 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-envoy\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281111 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-data\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281151 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-certs" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281164 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-socket\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281203 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-kube-api-access-nkknn\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-kube-api-access-nkknn\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281229 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-envoy\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281256 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf245911-b48a-4645-94e1-10a64054febd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf245911-b48a-4645-94e1-10a64054febd-istiod-ca-cert\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281264 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-data" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281286 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-credential-socket\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.281317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281289 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-socket" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.281838 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281346 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-podinfo\") pod \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\" (UID: \"b2bdecaa-9919-4802-b8f0-aaa24b2d0270\") " Nov 02 20:49:12.281838 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281409 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-istio-token\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-istio-token\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.281838 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281435 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-credential-socket\") pod \"cf245911-b48a-4645-94e1-10a64054febd\" (UID: \"cf245911-b48a-4645-94e1-10a64054febd\") " Nov 02 20:49:12.281838 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281195 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-certs" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.281838 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281665 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-credential-socket" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.282096 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281942 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.282096 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281962 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.282096 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281977 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.282096 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281974 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istiod-ca-cert" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:12.282096 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.281994 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.282096 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.282008 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.282388 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.282267 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-credential-socket" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.283699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.283449 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-kube-api-access-589q2" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "kube-api-access-589q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:12.283699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.283485 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-token" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:12.283859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.283718 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-data" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.283859 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.283848 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-kube-api-access-nkknn" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "kube-api-access-nkknn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:12.283981 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.283888 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-socket" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.284042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.284020 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-envoy" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.284291 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.284250 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf245911-b48a-4645-94e1-10a64054febd-istiod-ca-cert" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:12.284407 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.284376 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf245911-b48a-4645-94e1-10a64054febd-istio-podinfo" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:49:12.284494 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.284473 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-istio-token" pod "cf245911-b48a-4645-94e1-10a64054febd" (UID: "cf245911-b48a-4645-94e1-10a64054febd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:12.284576 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.284560 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-podinfo" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:49:12.284707 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.284665 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-envoy" pod "b2bdecaa-9919-4802-b8f0-aaa24b2d0270" (UID: "b2bdecaa-9919-4802-b8f0-aaa24b2d0270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:12.383137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383082 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383103 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383112 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383121 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383130 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf245911-b48a-4645-94e1-10a64054febd-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383138 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkknn\" (UniqueName: \"kubernetes.io/projected/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-kube-api-access-nkknn\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383149 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383156 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf245911-b48a-4645-94e1-10a64054febd-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383164 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383173 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2bdecaa-9919-4802-b8f0-aaa24b2d0270-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383181 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383189 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf245911-b48a-4645-94e1-10a64054febd-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.383380 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.383198 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-589q2\" (UniqueName: \"kubernetes.io/projected/cf245911-b48a-4645-94e1-10a64054febd-kube-api-access-589q2\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:12.779784 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.779626 5420 generic.go:356] "Generic (PLEG): container finished" podID="cf245911-b48a-4645-94e1-10a64054febd" containerID="ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9" exitCode=0 Nov 02 20:49:12.779784 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.779716 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" Nov 02 20:49:12.780295 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.779878 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" event={"ID":"cf245911-b48a-4645-94e1-10a64054febd","Type":"ContainerDied","Data":"ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9"} Nov 02 20:49:12.780295 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.779931 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n" event={"ID":"cf245911-b48a-4645-94e1-10a64054febd","Type":"ContainerDied","Data":"07b29134ee7f417fc7ff41366664fb166d684fb254915d148b0f46a94d6dbd42"} Nov 02 20:49:12.780295 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.779964 5420 scope.go:122] "RemoveContainer" containerID="ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9" Nov 02 20:49:12.785556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.783660 5420 generic.go:356] "Generic (PLEG): container finished" podID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerID="792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a" exitCode=0 Nov 02 20:49:12.785556 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.783884 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" Nov 02 20:49:12.790732 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.790687 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" event={"ID":"b2bdecaa-9919-4802-b8f0-aaa24b2d0270","Type":"ContainerDied","Data":"792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a"} Nov 02 20:49:12.790839 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.790746 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb" event={"ID":"b2bdecaa-9919-4802-b8f0-aaa24b2d0270","Type":"ContainerDied","Data":"4e79fcf5d4accdf678475b45e01ac3e2fed8bdc9e8290f563543690913b9eed6"} Nov 02 20:49:12.793838 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.793819 5420 scope.go:122] "RemoveContainer" containerID="ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9" Nov 02 20:49:12.794139 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:12.794120 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9\": container with ID starting with ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9 not found: ID does not exist" containerID="ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9" Nov 02 20:49:12.794220 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.794151 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9"} err="failed to get container status \"ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9\": rpc error: code = NotFound desc = could not find container \"ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9\": container with ID starting with ce036873137dd1ddf03ca56011fdc9820cdd767f05e3b5d96540b0b0bdf786e9 not found: ID does not exist" Nov 02 20:49:12.794220 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.794174 5420 scope.go:122] "RemoveContainer" containerID="792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a" Nov 02 20:49:12.803880 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.803812 5420 scope.go:122] "RemoveContainer" containerID="91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb" Nov 02 20:49:12.814511 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.814493 5420 scope.go:122] "RemoveContainer" containerID="792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a" Nov 02 20:49:12.814806 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:12.814784 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a\": container with ID starting with 792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a not found: ID does not exist" containerID="792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a" Nov 02 20:49:12.814873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.814813 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a"} err="failed to get container status \"792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a\": rpc error: code = NotFound desc = could not find container \"792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a\": container with ID starting with 792ea0ebcc27ecacf158fba9a2253d45f57484b38119cfb283ed2d5f3244d37a not found: ID does not exist" Nov 02 20:49:12.814873 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.814832 5420 scope.go:122] "RemoveContainer" containerID="91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb" Nov 02 20:49:12.815115 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:12.815087 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb\": container with ID starting with 91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb not found: ID does not exist" containerID="91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb" Nov 02 20:49:12.815194 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.815114 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb"} err="failed to get container status \"91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb\": rpc error: code = NotFound desc = could not find container \"91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb\": container with ID starting with 91b168395317833fb8c2067b745601de210822ea2cdc4b22e3e4ca5a65d328fb not found: ID does not exist" Nov 02 20:49:12.928474 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.928449 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:12.934471 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.934448 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n"] Nov 02 20:49:12.938185 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.938163 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-ckh4n"] Nov 02 20:49:12.991953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.991934 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-xks-ca\") pod \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " Nov 02 20:49:12.992147 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.992132 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-openshift-ca\") pod \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " Nov 02 20:49:12.992196 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.992165 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-kube-api-access-x5gzs\" (UniqueName: \"kubernetes.io/projected/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-kube-api-access-x5gzs\") pod \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\" (UID: \"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf\") " Nov 02 20:49:12.992522 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.992487 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-openshift-ca" pod "8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf" (UID: "8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:12.994069 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.994046 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-kube-api-access-x5gzs" pod "8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf" (UID: "8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf"). InnerVolumeSpecName "kube-api-access-x5gzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:12.994069 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:12.994044 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-xks-ca" pod "8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf" (UID: "8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:13.040414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.040370 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb"] Nov 02 20:49:13.046145 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.046126 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2250b4-openshift-default-6d6f4b6d7c-hbktb"] Nov 02 20:49:13.093124 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.093091 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-openshift-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.093124 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.093121 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5gzs\" (UniqueName: \"kubernetes.io/projected/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-kube-api-access-x5gzs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.093262 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.093138 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf-maas-api-xks-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.236632 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.236606 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2"] Nov 02 20:49:13.574197 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.574166 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" containerID="cri-o://f3c6d651f87c4f29d399ebb3a952051ba69f78bd5d426207d779c339012106cd" gracePeriod=45 Nov 02 20:49:13.595059 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.595031 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" containerID="cri-o://91702de1c8da01f920b0324ebaf4359bca285f9bab6a121756add26a185707b3" gracePeriod=45 Nov 02 20:49:13.775282 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.775246 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" path="/var/lib/kubelet/pods/b2bdecaa-9919-4802-b8f0-aaa24b2d0270/volumes" Nov 02 20:49:13.775875 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.775856 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf245911-b48a-4645-94e1-10a64054febd" path="/var/lib/kubelet/pods/cf245911-b48a-4645-94e1-10a64054febd/volumes" Nov 02 20:49:13.793885 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.793855 5420 generic.go:356] "Generic (PLEG): container finished" podID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerID="91702de1c8da01f920b0324ebaf4359bca285f9bab6a121756add26a185707b3" exitCode=0 Nov 02 20:49:13.794277 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.793929 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" event={"ID":"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6","Type":"ContainerDied","Data":"91702de1c8da01f920b0324ebaf4359bca285f9bab6a121756add26a185707b3"} Nov 02 20:49:13.795595 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.795569 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" event={"ID":"8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf","Type":"ContainerDied","Data":"b756901162f17bcf18909411015cdda1fbff1be18346c4c9e435d200cfb56489"} Nov 02 20:49:13.795752 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.795618 5420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b756901162f17bcf18909411015cdda1fbff1be18346c4c9e435d200cfb56489" Nov 02 20:49:13.795752 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.795721 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc" Nov 02 20:49:13.798796 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.798774 5420 generic.go:356] "Generic (PLEG): container finished" podID="ada702ce-5647-4d89-b72e-4af927326568" containerID="f3c6d651f87c4f29d399ebb3a952051ba69f78bd5d426207d779c339012106cd" exitCode=0 Nov 02 20:49:13.798900 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.798846 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" event={"ID":"ada702ce-5647-4d89-b72e-4af927326568","Type":"ContainerDied","Data":"f3c6d651f87c4f29d399ebb3a952051ba69f78bd5d426207d779c339012106cd"} Nov 02 20:49:13.853441 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.853412 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:49:13.869964 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.869942 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:49:13.901692 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.901659 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-model-cache\") pod \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " Nov 02 20:49:13.901853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.901752 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-dshm\") pod \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " Nov 02 20:49:13.901853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.901778 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada702ce-5647-4d89-b72e-4af927326568-kube-api-access-mpj4s\" (UniqueName: \"kubernetes.io/projected/ada702ce-5647-4d89-b72e-4af927326568-kube-api-access-mpj4s\") pod \"ada702ce-5647-4d89-b72e-4af927326568\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " Nov 02 20:49:13.901853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.901799 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-home\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-home\") pod \"ada702ce-5647-4d89-b72e-4af927326568\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " Nov 02 20:49:13.901853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.901834 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-dshm\") pod \"ada702ce-5647-4d89-b72e-4af927326568\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " Nov 02 20:49:13.902079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.901865 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-home\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-home\") pod \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " Nov 02 20:49:13.902079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.901936 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tls-certs\") pod \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " Nov 02 20:49:13.902079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.901945 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-model-cache" pod "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" (UID: "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.902079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902017 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tmp-dir\") pod \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " Nov 02 20:49:13.902079 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902047 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ada702ce-5647-4d89-b72e-4af927326568-tls-certs\" (UniqueName: \"kubernetes.io/secret/ada702ce-5647-4d89-b72e-4af927326568-tls-certs\") pod \"ada702ce-5647-4d89-b72e-4af927326568\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " Nov 02 20:49:13.902340 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902098 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-model-cache\") pod \"ada702ce-5647-4d89-b72e-4af927326568\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " Nov 02 20:49:13.902340 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902128 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-tmp-dir\") pod \"ada702ce-5647-4d89-b72e-4af927326568\" (UID: \"ada702ce-5647-4d89-b72e-4af927326568\") " Nov 02 20:49:13.902340 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902175 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-kube-api-access-f4m26\" (UniqueName: \"kubernetes.io/projected/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-kube-api-access-f4m26\") pod \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\" (UID: \"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6\") " Nov 02 20:49:13.902340 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902263 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-home" pod "ada702ce-5647-4d89-b72e-4af927326568" (UID: "ada702ce-5647-4d89-b72e-4af927326568"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.902531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902361 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-home" pod "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" (UID: "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.902619 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902598 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.902689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902626 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.902689 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902642 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:13.902905 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.902881 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tmp-dir" pod "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" (UID: "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.903157 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.903137 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-model-cache" pod "ada702ce-5647-4d89-b72e-4af927326568" (UID: "ada702ce-5647-4d89-b72e-4af927326568"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.904212 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.904188 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-tmp-dir" pod "ada702ce-5647-4d89-b72e-4af927326568" (UID: "ada702ce-5647-4d89-b72e-4af927326568"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.904719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.904666 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-dshm" pod "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" (UID: "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:13.904719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.904696 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ada702ce-5647-4d89-b72e-4af927326568-tls-certs" pod "ada702ce-5647-4d89-b72e-4af927326568" (UID: "ada702ce-5647-4d89-b72e-4af927326568"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:13.904719 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.904711 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada702ce-5647-4d89-b72e-4af927326568-kube-api-access-mpj4s" pod "ada702ce-5647-4d89-b72e-4af927326568" (UID: "ada702ce-5647-4d89-b72e-4af927326568"). InnerVolumeSpecName "kube-api-access-mpj4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:13.904915 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.904888 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tls-certs" pod "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" (UID: "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:13.905063 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.905043 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-kube-api-access-f4m26" pod "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" (UID: "0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6"). InnerVolumeSpecName "kube-api-access-f4m26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:13.905151 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:13.905074 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-dshm" pod "ada702ce-5647-4d89-b72e-4af927326568" (UID: "ada702ce-5647-4d89-b72e-4af927326568"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:14.004155 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004120 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.004155 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004158 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpj4s\" (UniqueName: \"kubernetes.io/projected/ada702ce-5647-4d89-b72e-4af927326568-kube-api-access-mpj4s\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.004332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004173 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.004332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004188 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.004332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004227 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.004332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004243 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ada702ce-5647-4d89-b72e-4af927326568-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.004332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004257 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.004332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004273 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ada702ce-5647-4d89-b72e-4af927326568-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.004332 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.004288 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4m26\" (UniqueName: \"kubernetes.io/projected/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6-kube-api-access-f4m26\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:14.807512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.807485 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" Nov 02 20:49:14.807512 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.807493 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn" event={"ID":"0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6","Type":"ContainerDied","Data":"c60c70e203166b8cfecbf85269a435bd92478fa04cdfe43d02b6d3eefe2b1a5c"} Nov 02 20:49:14.808121 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.807535 5420 scope.go:122] "RemoveContainer" containerID="91702de1c8da01f920b0324ebaf4359bca285f9bab6a121756add26a185707b3" Nov 02 20:49:14.809107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.809073 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" event={"ID":"ada702ce-5647-4d89-b72e-4af927326568","Type":"ContainerDied","Data":"60eb67171cb7123cb4c72245b73422cf81a6b9b5c2ce1b04968ee400b86f128f"} Nov 02 20:49:14.809230 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.809081 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct" Nov 02 20:49:14.817199 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.817185 5420 scope.go:122] "RemoveContainer" containerID="f3c6d651f87c4f29d399ebb3a952051ba69f78bd5d426207d779c339012106cd" Nov 02 20:49:14.960662 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:14.960639 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct"] Nov 02 20:49:15.006827 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:15.006792 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/distinct-2-w6-a5cad4-kserve-65775f9c4b-s9zct"] Nov 02 20:49:15.068344 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:15.068276 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn"] Nov 02 20:49:15.074222 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:15.074204 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a5cad4/premium-w6-a5cad4-kserve-5889c85cc8-sk7wn"] Nov 02 20:49:15.773936 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:15.773885 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" path="/var/lib/kubelet/pods/0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6/volumes" Nov 02 20:49:15.774334 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:15.774320 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada702ce-5647-4d89-b72e-4af927326568" path="/var/lib/kubelet/pods/ada702ce-5647-4d89-b72e-4af927326568/volumes" Nov 02 20:49:20.281536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:20.281502 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx"] Nov 02 20:49:20.401930 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:20.401898 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5"] Nov 02 20:49:25.282648 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:25.282609 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" podUID="943db2e6-08cb-49d8-a8d9-f6abd8754b13" containerName="payload-processing" containerID="cri-o://4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6" gracePeriod=25 Nov 02 20:49:25.402542 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:25.402474 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" podUID="f495bc83-6739-4f1c-9aae-cd8e54042154" containerName="payload-pre-processing" containerID="cri-o://646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba" gracePeriod=25 Nov 02 20:49:28.256355 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.256314 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" containerID="cri-o://18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116" gracePeriod=45 Nov 02 20:49:28.423585 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.423551 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": dial tcp 10.133.0.75:8000: connect: connection refused" Nov 02 20:49:28.522171 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.522123 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:49:28.638532 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.638505 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-dshm\") pod \"680154c9-0736-41e8-87f2-48e18a51c1f6\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " Nov 02 20:49:28.638738 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.638550 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-model-cache\") pod \"680154c9-0736-41e8-87f2-48e18a51c1f6\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " Nov 02 20:49:28.638738 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.638597 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680154c9-0736-41e8-87f2-48e18a51c1f6-kube-api-access-nfvj5\" (UniqueName: \"kubernetes.io/projected/680154c9-0736-41e8-87f2-48e18a51c1f6-kube-api-access-nfvj5\") pod \"680154c9-0736-41e8-87f2-48e18a51c1f6\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " Nov 02 20:49:28.638738 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.638642 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-tmp-dir\") pod \"680154c9-0736-41e8-87f2-48e18a51c1f6\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " Nov 02 20:49:28.638738 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.638716 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-home\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-home\") pod \"680154c9-0736-41e8-87f2-48e18a51c1f6\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " Nov 02 20:49:28.639621 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.638782 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680154c9-0736-41e8-87f2-48e18a51c1f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/680154c9-0736-41e8-87f2-48e18a51c1f6-tls-certs\") pod \"680154c9-0736-41e8-87f2-48e18a51c1f6\" (UID: \"680154c9-0736-41e8-87f2-48e18a51c1f6\") " Nov 02 20:49:28.640278 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.640233 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-model-cache" pod "680154c9-0736-41e8-87f2-48e18a51c1f6" (UID: "680154c9-0736-41e8-87f2-48e18a51c1f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.640547 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.640468 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-home" pod "680154c9-0736-41e8-87f2-48e18a51c1f6" (UID: "680154c9-0736-41e8-87f2-48e18a51c1f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.640996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.640971 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-tmp-dir" pod "680154c9-0736-41e8-87f2-48e18a51c1f6" (UID: "680154c9-0736-41e8-87f2-48e18a51c1f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.641914 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.641760 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680154c9-0736-41e8-87f2-48e18a51c1f6-tls-certs" pod "680154c9-0736-41e8-87f2-48e18a51c1f6" (UID: "680154c9-0736-41e8-87f2-48e18a51c1f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:28.642368 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.642344 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-dshm" pod "680154c9-0736-41e8-87f2-48e18a51c1f6" (UID: "680154c9-0736-41e8-87f2-48e18a51c1f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:28.647073 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.647051 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680154c9-0736-41e8-87f2-48e18a51c1f6-kube-api-access-nfvj5" pod "680154c9-0736-41e8-87f2-48e18a51c1f6" (UID: "680154c9-0736-41e8-87f2-48e18a51c1f6"). InnerVolumeSpecName "kube-api-access-nfvj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:28.740103 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.740073 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvj5\" (UniqueName: \"kubernetes.io/projected/680154c9-0736-41e8-87f2-48e18a51c1f6-kube-api-access-nfvj5\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.740103 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.740104 5420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-tmp-dir\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.740287 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.740115 5420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-home\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.740287 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.740123 5420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680154c9-0736-41e8-87f2-48e18a51c1f6-tls-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.740287 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.740131 5420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-dshm\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.740287 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.740139 5420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680154c9-0736-41e8-87f2-48e18a51c1f6-model-cache\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:28.892029 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.891998 5420 generic.go:356] "Generic (PLEG): container finished" podID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerID="18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116" exitCode=0 Nov 02 20:49:28.892207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.892043 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" Nov 02 20:49:28.892207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.892052 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" event={"ID":"680154c9-0736-41e8-87f2-48e18a51c1f6","Type":"ContainerDied","Data":"18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116"} Nov 02 20:49:28.892207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.892081 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2" event={"ID":"680154c9-0736-41e8-87f2-48e18a51c1f6","Type":"ContainerDied","Data":"2ff99f8f29674261229eeb0810d0dd910804a701add6851d6e3b7a7d9151a8ec"} Nov 02 20:49:28.892207 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.892097 5420 scope.go:122] "RemoveContainer" containerID="18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116" Nov 02 20:49:28.903248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.903234 5420 scope.go:122] "RemoveContainer" containerID="18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116" Nov 02 20:49:28.903520 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:28.903502 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116\": container with ID starting with 18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116 not found: ID does not exist" containerID="18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116" Nov 02 20:49:28.903596 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:28.903527 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116"} err="failed to get container status \"18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116\": rpc error: code = NotFound desc = could not find container \"18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116\": container with ID starting with 18cabd595e46dd874d12be66c2949dbea5441fc3ae06293cb6c14edb635b3116 not found: ID does not exist" Nov 02 20:49:29.048231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:29.048201 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2"] Nov 02 20:49:29.094524 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:29.094493 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-61cdc1/distinct-2-w3-61cdc1-kserve-6fd6cf888c-9kmj2"] Nov 02 20:49:29.774605 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:29.774573 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" path="/var/lib/kubelet/pods/680154c9-0736-41e8-87f2-48e18a51c1f6/volumes" Nov 02 20:49:30.096493 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.096460 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq"] Nov 02 20:49:30.096729 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.096706 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" podUID="cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" containerName="maas-api" containerID="cri-o://5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4" gracePeriod=30 Nov 02 20:49:30.349525 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.349468 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:49:30.458492 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.458464 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls\") pod \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " Nov 02 20:49:30.458636 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.458502 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-metrics-certs\") pod \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " Nov 02 20:49:30.458636 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.458563 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-openshift-service-ca\") pod \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " Nov 02 20:49:30.458636 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.458599 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-kube-api-access-zlbjr\" (UniqueName: \"kubernetes.io/projected/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-kube-api-access-zlbjr\") pod \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\" (UID: \"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f\") " Nov 02 20:49:30.459280 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.459199 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-openshift-service-ca" pod "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" (UID: "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:30.460942 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.460907 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls" pod "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" (UID: "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:30.461046 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.460976 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-kube-api-access-zlbjr" pod "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" (UID: "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f"). InnerVolumeSpecName "kube-api-access-zlbjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:30.461304 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.461282 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-metrics-certs" pod "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" (UID: "cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:49:30.559957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.559930 5420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-openshift-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:30.559957 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.559955 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlbjr\" (UniqueName: \"kubernetes.io/projected/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-kube-api-access-zlbjr\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:30.560093 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.559964 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-maas-api-tls\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:30.560093 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.559973 5420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f-metrics-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:30.905653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.905621 5420 generic.go:356] "Generic (PLEG): container finished" podID="cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" containerID="5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4" exitCode=0 Nov 02 20:49:30.905653 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.905655 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" event={"ID":"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f","Type":"ContainerDied","Data":"5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4"} Nov 02 20:49:30.906197 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.905697 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" Nov 02 20:49:30.906197 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.905714 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq" event={"ID":"cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f","Type":"ContainerDied","Data":"24b3a5033bf8839e543258a130817a9859aeacf3ce0458aee428ee1787b823d2"} Nov 02 20:49:30.906197 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.905741 5420 scope.go:122] "RemoveContainer" containerID="5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4" Nov 02 20:49:30.921123 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.921104 5420 scope.go:122] "RemoveContainer" containerID="5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4" Nov 02 20:49:30.921403 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:30.921380 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4\": container with ID starting with 5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4 not found: ID does not exist" containerID="5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4" Nov 02 20:49:30.921461 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:30.921416 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4"} err="failed to get container status \"5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4\": rpc error: code = NotFound desc = could not find container \"5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4\": container with ID starting with 5850c59d61ad72d9ae1bf98f3d0e0af12bd68a85d87f7eeeb470f1b12026b5b4 not found: ID does not exist" Nov 02 20:49:31.058205 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:31.058175 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq"] Nov 02 20:49:31.061905 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:31.061880 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a5cad4-686984d57-fh8hq"] Nov 02 20:49:31.775043 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:31.775001 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" path="/var/lib/kubelet/pods/cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f/volumes" Nov 02 20:49:32.657118 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:32.657081 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:49:34.427991 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:34.427955 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" podUID="f495bc83-6739-4f1c-9aae-cd8e54042154" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:49:34.428387 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:34.427951 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" podUID="943db2e6-08cb-49d8-a8d9-f6abd8754b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:49:44.428783 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:44.428745 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" podUID="943db2e6-08cb-49d8-a8d9-f6abd8754b13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:49:44.429184 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:44.428745 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" podUID="f495bc83-6739-4f1c-9aae-cd8e54042154" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 20:49:45.525774 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.525754 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:49:45.627416 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.627392 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:49:45.692465 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.692436 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943db2e6-08cb-49d8-a8d9-f6abd8754b13-kube-api-access-l4sw4\" (UniqueName: \"kubernetes.io/projected/943db2e6-08cb-49d8-a8d9-f6abd8754b13-kube-api-access-l4sw4\") pod \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\" (UID: \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\") " Nov 02 20:49:45.692591 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.692576 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943db2e6-08cb-49d8-a8d9-f6abd8754b13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/943db2e6-08cb-49d8-a8d9-f6abd8754b13-plugins-config-volume\") pod \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\" (UID: \"943db2e6-08cb-49d8-a8d9-f6abd8754b13\") " Nov 02 20:49:45.692971 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.692946 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943db2e6-08cb-49d8-a8d9-f6abd8754b13-plugins-config-volume" pod "943db2e6-08cb-49d8-a8d9-f6abd8754b13" (UID: "943db2e6-08cb-49d8-a8d9-f6abd8754b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:45.694534 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.694514 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943db2e6-08cb-49d8-a8d9-f6abd8754b13-kube-api-access-l4sw4" pod "943db2e6-08cb-49d8-a8d9-f6abd8754b13" (UID: "943db2e6-08cb-49d8-a8d9-f6abd8754b13"). InnerVolumeSpecName "kube-api-access-l4sw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:45.793237 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.793213 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f495bc83-6739-4f1c-9aae-cd8e54042154-kube-api-access-zqst8\" (UniqueName: \"kubernetes.io/projected/f495bc83-6739-4f1c-9aae-cd8e54042154-kube-api-access-zqst8\") pod \"f495bc83-6739-4f1c-9aae-cd8e54042154\" (UID: \"f495bc83-6739-4f1c-9aae-cd8e54042154\") " Nov 02 20:49:45.793354 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.793268 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f495bc83-6739-4f1c-9aae-cd8e54042154-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f495bc83-6739-4f1c-9aae-cd8e54042154-plugins-config-volume\") pod \"f495bc83-6739-4f1c-9aae-cd8e54042154\" (UID: \"f495bc83-6739-4f1c-9aae-cd8e54042154\") " Nov 02 20:49:45.793438 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.793427 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/943db2e6-08cb-49d8-a8d9-f6abd8754b13-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:45.793479 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.793440 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4sw4\" (UniqueName: \"kubernetes.io/projected/943db2e6-08cb-49d8-a8d9-f6abd8754b13-kube-api-access-l4sw4\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:45.793619 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.793595 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f495bc83-6739-4f1c-9aae-cd8e54042154-plugins-config-volume" pod "f495bc83-6739-4f1c-9aae-cd8e54042154" (UID: "f495bc83-6739-4f1c-9aae-cd8e54042154"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:45.795172 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.795153 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f495bc83-6739-4f1c-9aae-cd8e54042154-kube-api-access-zqst8" pod "f495bc83-6739-4f1c-9aae-cd8e54042154" (UID: "f495bc83-6739-4f1c-9aae-cd8e54042154"). InnerVolumeSpecName "kube-api-access-zqst8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:45.894397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.894343 5420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f495bc83-6739-4f1c-9aae-cd8e54042154-plugins-config-volume\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:45.894397 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.894369 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqst8\" (UniqueName: \"kubernetes.io/projected/f495bc83-6739-4f1c-9aae-cd8e54042154-kube-api-access-zqst8\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:45.994164 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.994139 5420 generic.go:356] "Generic (PLEG): container finished" podID="943db2e6-08cb-49d8-a8d9-f6abd8754b13" containerID="4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6" exitCode=0 Nov 02 20:49:45.994279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.994191 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" Nov 02 20:49:45.994279 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.994232 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" event={"ID":"943db2e6-08cb-49d8-a8d9-f6abd8754b13","Type":"ContainerDied","Data":"4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6"} Nov 02 20:49:45.994385 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.994283 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx" event={"ID":"943db2e6-08cb-49d8-a8d9-f6abd8754b13","Type":"ContainerDied","Data":"f49b904d1fae68f5500a55fbb2e264fa09b68d660282b53676f0e4c64f0aa5a6"} Nov 02 20:49:45.994385 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.994314 5420 scope.go:122] "RemoveContainer" containerID="4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6" Nov 02 20:49:45.996276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.996254 5420 generic.go:356] "Generic (PLEG): container finished" podID="f495bc83-6739-4f1c-9aae-cd8e54042154" containerID="646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba" exitCode=0 Nov 02 20:49:45.996409 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.996394 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" event={"ID":"f495bc83-6739-4f1c-9aae-cd8e54042154","Type":"ContainerDied","Data":"646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba"} Nov 02 20:49:45.996475 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.996434 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" event={"ID":"f495bc83-6739-4f1c-9aae-cd8e54042154","Type":"ContainerDied","Data":"7b8f82546acdcc1dafe5f3c590ba7a06435ee40b1aa2c0cc8abc8db9f2bc25f9"} Nov 02 20:49:45.996536 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:45.996513 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5" Nov 02 20:49:46.004408 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.004391 5420 scope.go:122] "RemoveContainer" containerID="4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6" Nov 02 20:49:46.004628 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:46.004610 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6\": container with ID starting with 4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6 not found: ID does not exist" containerID="4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6" Nov 02 20:49:46.004741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.004639 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6"} err="failed to get container status \"4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6\": rpc error: code = NotFound desc = could not find container \"4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6\": container with ID starting with 4a8884c8e9d5efdfda13f871bc89099dae5d1f81bfaac0b3b7ac24c0fc1cbde6 not found: ID does not exist" Nov 02 20:49:46.004741 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.004660 5420 scope.go:122] "RemoveContainer" containerID="646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba" Nov 02 20:49:46.013042 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.013026 5420 scope.go:122] "RemoveContainer" containerID="646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba" Nov 02 20:49:46.013276 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:46.013259 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba\": container with ID starting with 646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba not found: ID does not exist" containerID="646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba" Nov 02 20:49:46.013327 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.013284 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba"} err="failed to get container status \"646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba\": rpc error: code = NotFound desc = could not find container \"646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba\": container with ID starting with 646e134d563745f3edc715d5827ca587eae13c75e9be53151159e805c9b016ba not found: ID does not exist" Nov 02 20:49:46.147148 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.147093 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx"] Nov 02 20:49:46.192852 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.192832 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-61cdc1-bf567dc8d-n9whx"] Nov 02 20:49:46.252330 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.252310 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5"] Nov 02 20:49:46.258372 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:46.258355 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-61cdc1-ccc9c9ccf-qbnt5"] Nov 02 20:49:47.778772 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:47.778735 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943db2e6-08cb-49d8-a8d9-f6abd8754b13" path="/var/lib/kubelet/pods/943db2e6-08cb-49d8-a8d9-f6abd8754b13/volumes" Nov 02 20:49:47.781342 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:47.779218 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f495bc83-6739-4f1c-9aae-cd8e54042154" path="/var/lib/kubelet/pods/f495bc83-6739-4f1c-9aae-cd8e54042154/volumes" Nov 02 20:49:48.031421 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:48.031340 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4"] Nov 02 20:49:48.031604 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:48.031542 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" containerID="cri-o://4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2" gracePeriod=30 Nov 02 20:49:50.663684 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:50.663635 5420 patch_prober.go:29] interesting pod/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:49:50.664050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:50.663706 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:49:53.281628 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.281604 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:49:53.360702 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.360665 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-podinfo\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.360832 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.360721 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istiod-ca-cert\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.360832 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.360797 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-kube-api-access-xvtb9\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-kube-api-access-xvtb9\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.360951 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.360840 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-token\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-token\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.360951 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.360872 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-credential-socket\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.360951 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.360911 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-certs\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.361107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.360952 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-envoy\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.361107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.360993 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-socket\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.361107 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.361021 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-data\") pod \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\" (UID: \"a5b79a65-9be4-4acb-b68b-19dca7de84cd\") " Nov 02 20:49:53.361282 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.361172 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-credential-socket" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:53.361282 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.361208 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-certs" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:53.361412 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.361388 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-data" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:53.361412 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.361402 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-socket" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:53.361523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.361439 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istiod-ca-cert" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:49:53.361523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.361459 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:53.361523 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.361477 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:53.362920 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.362896 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-podinfo" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:49:53.362920 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.362896 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-kube-api-access-xvtb9" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "kube-api-access-xvtb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:53.363342 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.363314 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-token" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:49:53.363433 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.363348 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-envoy" pod "a5b79a65-9be4-4acb-b68b-19dca7de84cd" (UID: "a5b79a65-9be4-4acb-b68b-19dca7de84cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:49:53.462996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.462948 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvtb9\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-kube-api-access-xvtb9\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:53.462996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.462982 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:53.462996 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.462996 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:53.463264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.463008 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:53.463264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.463023 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:53.463264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.463034 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:53.463264 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:53.463046 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5b79a65-9be4-4acb-b68b-19dca7de84cd-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:49:54.049390 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.049362 5420 generic.go:356] "Generic (PLEG): container finished" podID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerID="4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2" exitCode=0 Nov 02 20:49:54.049549 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.049413 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" Nov 02 20:49:54.049549 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.049428 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" event={"ID":"a5b79a65-9be4-4acb-b68b-19dca7de84cd","Type":"ContainerDied","Data":"4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2"} Nov 02 20:49:54.049549 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.049487 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4" event={"ID":"a5b79a65-9be4-4acb-b68b-19dca7de84cd","Type":"ContainerDied","Data":"e42db50a8de7fe5e967ecb8d3c9ab323aafae41bd8a262c01f8921a1348f77ba"} Nov 02 20:49:54.049549 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.049511 5420 scope.go:122] "RemoveContainer" containerID="4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2" Nov 02 20:49:54.060144 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.060127 5420 scope.go:122] "RemoveContainer" containerID="deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889" Nov 02 20:49:54.068823 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.068806 5420 scope.go:122] "RemoveContainer" containerID="4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2" Nov 02 20:49:54.069121 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:54.069094 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2\": container with ID starting with 4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2 not found: ID does not exist" containerID="4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2" Nov 02 20:49:54.069201 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.069134 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2"} err="failed to get container status \"4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2\": rpc error: code = NotFound desc = could not find container \"4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2\": container with ID starting with 4fd0ae973eb9f4c05c4227b4f9fcccc8bd5bfcdc00f4a6112ead5379057c1aa2 not found: ID does not exist" Nov 02 20:49:54.069201 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.069158 5420 scope.go:122] "RemoveContainer" containerID="deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889" Nov 02 20:49:54.069410 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:49:54.069392 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889\": container with ID starting with deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889 not found: ID does not exist" containerID="deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889" Nov 02 20:49:54.069459 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.069417 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889"} err="failed to get container status \"deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889\": rpc error: code = NotFound desc = could not find container \"deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889\": container with ID starting with deabae815b2c141350df47057c68c09f6e3ead54d12626e734253d23991f2889 not found: ID does not exist" Nov 02 20:49:54.250415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.250389 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4"] Nov 02 20:49:54.296724 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:54.296699 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-61cdc1-openshift-default-64d5949c5f-fhfd4"] Nov 02 20:49:55.773586 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:49:55.773554 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" path="/var/lib/kubelet/pods/a5b79a65-9be4-4acb-b68b-19dca7de84cd/volumes" Nov 02 20:50:02.657081 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:50:02.657044 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:50:32.656426 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:50:32.656388 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:50:41.558651 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.558615 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g"] Nov 02 20:50:41.559221 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559200 5420 cpu_manager.go:414] "containerMap: removing container" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" Nov 02 20:50:41.559309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559226 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" Nov 02 20:50:41.559309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559262 5420 cpu_manager.go:414] "containerMap: removing container" podUID="943db2e6-08cb-49d8-a8d9-f6abd8754b13" containerName="payload-processing" Nov 02 20:50:41.559309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559270 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943db2e6-08cb-49d8-a8d9-f6abd8754b13" containerName="payload-processing" Nov 02 20:50:41.559309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559286 5420 cpu_manager.go:414] "containerMap: removing container" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" Nov 02 20:50:41.559309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559294 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c61ef80-a2e5-4319-8c6f-ba79be60bdf3" containerName="istio-proxy" Nov 02 20:50:41.559309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559307 5420 cpu_manager.go:414] "containerMap: removing container" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559315 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559327 5420 cpu_manager.go:414] "containerMap: removing container" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559345 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559355 5420 cpu_manager.go:414] "containerMap: removing container" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559362 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559376 5420 cpu_manager.go:414] "containerMap: removing container" podUID="8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf" containerName="revoke-keys" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559384 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf" containerName="revoke-keys" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559396 5420 cpu_manager.go:414] "containerMap: removing container" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559403 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559416 5420 cpu_manager.go:414] "containerMap: removing container" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559422 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559430 5420 cpu_manager.go:414] "containerMap: removing container" podUID="f495bc83-6739-4f1c-9aae-cd8e54042154" containerName="payload-pre-processing" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559436 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f495bc83-6739-4f1c-9aae-cd8e54042154" containerName="payload-pre-processing" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559457 5420 cpu_manager.go:414] "containerMap: removing container" podUID="cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" containerName="maas-api" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559464 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" containerName="maas-api" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559488 5420 cpu_manager.go:414] "containerMap: removing container" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559496 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" Nov 02 20:50:41.559611 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559609 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559623 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="680154c9-0736-41e8-87f2-48e18a51c1f6" containerName="main" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559637 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada702ce-5647-4d89-b72e-4af927326568" containerName="main" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559651 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="035c4f0f-4f92-4ca6-8e3a-bc84ef17e73d" containerName="istio-proxy" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559663 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="943db2e6-08cb-49d8-a8d9-f6abd8754b13" containerName="payload-processing" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559686 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559696 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf245911-b48a-4645-94e1-10a64054febd" containerName="istio-proxy" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559704 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf" containerName="revoke-keys" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559713 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6bdb14-0bc7-4c61-856c-38f9ec36a9a6" containerName="main" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559722 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bdecaa-9919-4802-b8f0-aaa24b2d0270" containerName="istio-proxy" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559729 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f495bc83-6739-4f1c-9aae-cd8e54042154" containerName="payload-pre-processing" Nov 02 20:50:41.560498 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.559738 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc6e351-3134-45b1-88cc-ae8cfd1f9c8f" containerName="maas-api" Nov 02 20:50:41.564843 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.564822 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.662307 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.662281 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g"] Nov 02 20:50:41.667832 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.667804 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.667962 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.667924 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.668013 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.667973 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4q5\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-kube-api-access-xk4q5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.668059 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.668014 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.668193 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.668168 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.668245 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.668229 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.668317 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.668299 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.668360 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.668331 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.668414 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.668361 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.769785 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.769756 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.769953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.769818 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.769953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.769839 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4q5\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-kube-api-access-xk4q5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.769953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.769870 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.769953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.769905 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.769953 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.769947 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.770231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.769976 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.770231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.770008 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.770231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.770038 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.770231 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.770220 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.770447 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.770430 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.770505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.770467 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.770693 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.770625 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.771020 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.770996 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.772818 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.772795 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.772928 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.772862 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.827424 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.827364 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.827545 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.827530 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4q5\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-kube-api-access-xk4q5\") pod \"maas-default-gateway-openshift-default-85544d4cd7-qfm4g\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:41.877438 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:41.877418 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:42.291090 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:42.291067 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g"] Nov 02 20:50:42.293056 ip-10-0-134-206 kubenswrapper[5420]: W1102 20:50:42.293027 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a9c98d_5ec6_4e6d_9178_6bb06f51888e.slice/crio-67f8abec0bb1d73c274aede3158b4f1fadfac402f1d3d08eaa0dca2192622889 WatchSource:0}: Error finding container 67f8abec0bb1d73c274aede3158b4f1fadfac402f1d3d08eaa0dca2192622889: Status 404 returned error can't find the container with id 67f8abec0bb1d73c274aede3158b4f1fadfac402f1d3d08eaa0dca2192622889 Nov 02 20:50:42.295191 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:42.295162 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:50:42.295276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:42.295224 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:50:42.295276 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:42.295252 5420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 20:50:42.316222 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:42.316200 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" event={"ID":"69a9c98d-5ec6-4e6d-9178-6bb06f51888e","Type":"ContainerStarted","Data":"67f8abec0bb1d73c274aede3158b4f1fadfac402f1d3d08eaa0dca2192622889"} Nov 02 20:50:43.323941 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:43.323904 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" event={"ID":"69a9c98d-5ec6-4e6d-9178-6bb06f51888e","Type":"ContainerStarted","Data":"aa40761fa46f152a2dc9f60c4681764e52cee1f3c385c02d4ea3631f2dfa29d3"} Nov 02 20:50:43.533533 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:43.533462 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" podStartSLOduration=2.53344813 podStartE2EDuration="2.53344813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 20:50:43.487045709 +0000 UTC m=+2115.851656914" watchObservedRunningTime="2026-11-02 20:50:43.53344813 +0000 UTC m=+2115.898059372" Nov 02 20:50:43.879488 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:43.879457 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:43.879828 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:43.879801 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:50:43.880007 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:43.879983 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 20:50:44.877988 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:44.877951 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:50:44.878563 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:44.878532 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 20:50:45.882981 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:45.882947 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:50:45.883375 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:45.882998 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 20:50:46.881987 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:46.881949 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:50:46.882273 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:46.882244 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 20:50:47.878014 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:47.877977 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:50:47.878610 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:47.878579 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 20:50:48.878141 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:48.878098 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-qfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 02 20:50:48.878569 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:48.878170 5420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 02 20:50:50.317541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:50.317501 5420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:50.318040 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:50.317935 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:50:50.318932 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:50:50.318912 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:51:02.657977 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:51:02.657939 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:51:32.656501 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:51:32.656466 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:52:02.657906 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:52:02.657819 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:52:10.950896 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:10.950867 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj"] Nov 02 20:52:10.951405 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:10.951045 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" podUID="91a3c81d-7baa-4a47-b018-931d8e75c077" containerName="discovery" containerID="cri-o://3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734" gracePeriod=30 Nov 02 20:52:11.202636 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.202587 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:52:11.291875 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.291851 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-token\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-token\") pod \"91a3c81d-7baa-4a47-b018-931d8e75c077\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " Nov 02 20:52:11.291971 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.291892 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-dns-cert\") pod \"91a3c81d-7baa-4a47-b018-931d8e75c077\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " Nov 02 20:52:11.291971 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.291921 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-kubeconfig\") pod \"91a3c81d-7baa-4a47-b018-931d8e75c077\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " Nov 02 20:52:11.292050 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.292033 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-cacerts\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-cacerts\") pod \"91a3c81d-7baa-4a47-b018-931d8e75c077\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " Nov 02 20:52:11.292103 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.292089 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-ca-configmap\") pod \"91a3c81d-7baa-4a47-b018-931d8e75c077\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " Nov 02 20:52:11.292225 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.292201 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91a3c81d-7baa-4a47-b018-931d8e75c077-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a3c81d-7baa-4a47-b018-931d8e75c077-local-certs\") pod \"91a3c81d-7baa-4a47-b018-931d8e75c077\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " Nov 02 20:52:11.292309 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.292274 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-kube-api-access-h9hwn\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-kube-api-access-h9hwn\") pod \"91a3c81d-7baa-4a47-b018-931d8e75c077\" (UID: \"91a3c81d-7baa-4a47-b018-931d8e75c077\") " Nov 02 20:52:11.292662 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.292514 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-ca-configmap" pod "91a3c81d-7baa-4a47-b018-931d8e75c077" (UID: "91a3c81d-7baa-4a47-b018-931d8e75c077"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:52:11.292662 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.292632 5420 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-ca-configmap\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:11.294100 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.294075 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-kubeconfig" pod "91a3c81d-7baa-4a47-b018-931d8e75c077" (UID: "91a3c81d-7baa-4a47-b018-931d8e75c077"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:52:11.294248 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.294226 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-cacerts" pod "91a3c81d-7baa-4a47-b018-931d8e75c077" (UID: "91a3c81d-7baa-4a47-b018-931d8e75c077"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:52:11.294382 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.294354 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-token" pod "91a3c81d-7baa-4a47-b018-931d8e75c077" (UID: "91a3c81d-7baa-4a47-b018-931d8e75c077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:11.294475 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.294379 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-dns-cert" pod "91a3c81d-7baa-4a47-b018-931d8e75c077" (UID: "91a3c81d-7baa-4a47-b018-931d8e75c077"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 20:52:11.294683 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.294642 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91a3c81d-7baa-4a47-b018-931d8e75c077-local-certs" pod "91a3c81d-7baa-4a47-b018-931d8e75c077" (UID: "91a3c81d-7baa-4a47-b018-931d8e75c077"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:52:11.294770 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.294710 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-kube-api-access-h9hwn" pod "91a3c81d-7baa-4a47-b018-931d8e75c077" (UID: "91a3c81d-7baa-4a47-b018-931d8e75c077"). InnerVolumeSpecName "kube-api-access-h9hwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:52:11.393853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.393825 5420 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/91a3c81d-7baa-4a47-b018-931d8e75c077-local-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:11.393853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.393847 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hwn\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-kube-api-access-h9hwn\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:11.393853 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.393860 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:11.394061 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.393868 5420 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-csr-dns-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:11.394061 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.393877 5420 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-istio-kubeconfig\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:11.394061 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.393885 5420 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/91a3c81d-7baa-4a47-b018-931d8e75c077-cacerts\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:52:11.788418 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.788385 5420 generic.go:356] "Generic (PLEG): container finished" podID="91a3c81d-7baa-4a47-b018-931d8e75c077" containerID="3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734" exitCode=0 Nov 02 20:52:11.788531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.788429 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" Nov 02 20:52:11.788531 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.788465 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" event={"ID":"91a3c81d-7baa-4a47-b018-931d8e75c077","Type":"ContainerDied","Data":"3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734"} Nov 02 20:52:11.788618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.788533 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj" event={"ID":"91a3c81d-7baa-4a47-b018-931d8e75c077","Type":"ContainerDied","Data":"c1ab2a1d966dc3521a6e3b209a812e44baf7345fce659fc03bb7f18f9709b8cf"} Nov 02 20:52:11.788618 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.788556 5420 scope.go:122] "RemoveContainer" containerID="3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734" Nov 02 20:52:11.800550 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.800522 5420 scope.go:122] "RemoveContainer" containerID="3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734" Nov 02 20:52:11.800925 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:52:11.800894 5420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734\": container with ID starting with 3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734 not found: ID does not exist" containerID="3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734" Nov 02 20:52:11.801053 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.800932 5420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734"} err="failed to get container status \"3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734\": rpc error: code = NotFound desc = could not find container \"3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734\": container with ID starting with 3840e501f4359c2038d16b0a36310c2f27a862ef388d856e3df94329a201c734 not found: ID does not exist" Nov 02 20:52:11.987557 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:11.987532 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj"] Nov 02 20:52:12.033864 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:12.033840 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zsbnj"] Nov 02 20:52:13.773476 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:52:13.773445 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91a3c81d-7baa-4a47-b018-931d8e75c077" path="/var/lib/kubelet/pods/91a3c81d-7baa-4a47-b018-931d8e75c077/volumes" Nov 02 20:52:32.656824 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:52:32.656782 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:53:02.656324 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:53:02.656287 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:53:32.657582 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:53:32.657497 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:54:02.656368 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:54:02.656334 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:54:14.048327 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:54:14.048284 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc"] Nov 02 20:54:14.094323 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:54:14.094297 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d935bb-8n4nc"] Nov 02 20:54:15.773009 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:54:15.772969 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf" path="/var/lib/kubelet/pods/8ca59f5f-cbe8-49a5-8a0a-80ad1b525baf/volumes" Nov 02 20:54:32.657610 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:54:32.657573 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:55:02.657137 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:55:02.657102 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:55:28.552606 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:55:28.552576 5420 scope.go:122] "RemoveContainer" containerID="e71d083b129cb1cee3e47d21d21bff55f28ef2600b5543601167b78c65d21bd9" Nov 02 20:55:32.657777 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:55:32.657731 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:55:58.748838 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:55:58.748805 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp"] Nov 02 20:55:58.749286 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:55:58.749008 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" containerID="cri-o://56dcd9f299e63dc3eb565ab28c7f2214099de50023d801984df20c8e062604fc" gracePeriod=30 Nov 02 20:55:58.795168 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:55:58.795091 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g"] Nov 02 20:55:58.795339 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:55:58.795321 5420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" containerID="cri-o://aa40761fa46f152a2dc9f60c4681764e52cee1f3c385c02d4ea3631f2dfa29d3" gracePeriod=30 Nov 02 20:56:02.656512 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:56:02.656465 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:56:02.741647 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:02.741610 5420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-zd2pp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 20:56:02.741845 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:02.741705 5420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 20:56:03.955118 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:03.955056 5420 generic.go:356] "Generic (PLEG): container finished" podID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerID="56dcd9f299e63dc3eb565ab28c7f2214099de50023d801984df20c8e062604fc" exitCode=0 Nov 02 20:56:03.955505 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:03.955208 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" event={"ID":"e44f81ce-47d4-4b30-ab50-90135d0b45d2","Type":"ContainerDied","Data":"56dcd9f299e63dc3eb565ab28c7f2214099de50023d801984df20c8e062604fc"} Nov 02 20:56:03.957270 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:03.957236 5420 generic.go:356] "Generic (PLEG): container finished" podID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerID="aa40761fa46f152a2dc9f60c4681764e52cee1f3c385c02d4ea3631f2dfa29d3" exitCode=0 Nov 02 20:56:03.957389 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:03.957281 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" event={"ID":"69a9c98d-5ec6-4e6d-9178-6bb06f51888e","Type":"ContainerDied","Data":"aa40761fa46f152a2dc9f60c4681764e52cee1f3c385c02d4ea3631f2dfa29d3"} Nov 02 20:56:04.031252 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.031231 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:56:04.064761 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.064740 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:56:04.106949 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.106927 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-certs\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.106960 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-socket\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.106982 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istiod-ca-cert\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107031 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-token\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-token\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107067 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107060 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-data\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107289 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107080 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-kube-api-access-6fzcx\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-kube-api-access-6fzcx\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107289 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107111 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-credential-socket\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107289 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107169 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-envoy\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107289 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107202 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-certs" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.107289 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107227 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-podinfo\") pod \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\" (UID: \"e44f81ce-47d4-4b30-ab50-90135d0b45d2\") " Nov 02 20:56:04.107289 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107268 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-socket" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.107586 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107503 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-credential-socket" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.107642 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107614 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-data" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.107699 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107651 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istiod-ca-cert" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:56:04.107744 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107662 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.107744 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107726 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.107845 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.107745 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.109387 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.109356 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-token" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:56:04.109488 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.109363 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-kube-api-access-6fzcx" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "kube-api-access-6fzcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:56:04.109541 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.109527 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-podinfo" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:56:04.109753 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.109737 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-envoy" pod "e44f81ce-47d4-4b30-ab50-90135d0b45d2" (UID: "e44f81ce-47d4-4b30-ab50-90135d0b45d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.208272 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208248 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-certs\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208286 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istiod-ca-cert\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208307 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-credential-socket\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208332 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-kube-api-access-xk4q5\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-kube-api-access-xk4q5\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208366 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-podinfo\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208415 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208400 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-socket\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208453 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-envoy\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208499 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-token\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-token\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208535 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-data\") pod \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\" (UID: \"69a9c98d-5ec6-4e6d-9178-6bb06f51888e\") " Nov 02 20:56:04.208731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208601 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-certs" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.208731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208694 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-credential-socket" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.208731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208722 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istiod-ca-cert" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 20:56:04.209066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208888 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-socket" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.209066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208921 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-data" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.209066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208972 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.209066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.208993 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fzcx\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-kube-api-access-6fzcx\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.209066 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.209010 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.209351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.209136 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.209351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.209155 5420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-certs\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.209351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.209171 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.209351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.209184 5420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-credential-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.209351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.209198 5420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istiod-ca-cert\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.209351 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.209211 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e44f81ce-47d4-4b30-ab50-90135d0b45d2-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.210869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.210832 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-podinfo" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 20:56:04.210869 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.210837 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-kube-api-access-xk4q5" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "kube-api-access-xk4q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:56:04.211134 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.211114 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-envoy" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 20:56:04.211247 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.211229 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-token" pod "69a9c98d-5ec6-4e6d-9178-6bb06f51888e" (UID: "69a9c98d-5ec6-4e6d-9178-6bb06f51888e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 20:56:04.310613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.310591 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4q5\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-kube-api-access-xk4q5\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.310613 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.310614 5420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-podinfo\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.310778 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.310624 5420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-workload-socket\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.310778 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.310632 5420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-envoy\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.310778 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.310641 5420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-token\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.310778 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.310648 5420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a9c98d-5ec6-4e6d-9178-6bb06f51888e-istio-data\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 20:56:04.963733 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.963704 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" Nov 02 20:56:04.964139 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.963726 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g" event={"ID":"69a9c98d-5ec6-4e6d-9178-6bb06f51888e","Type":"ContainerDied","Data":"67f8abec0bb1d73c274aede3158b4f1fadfac402f1d3d08eaa0dca2192622889"} Nov 02 20:56:04.964139 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.963794 5420 scope.go:122] "RemoveContainer" containerID="aa40761fa46f152a2dc9f60c4681764e52cee1f3c385c02d4ea3631f2dfa29d3" Nov 02 20:56:04.965115 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.965096 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" event={"ID":"e44f81ce-47d4-4b30-ab50-90135d0b45d2","Type":"ContainerDied","Data":"71983dfede3b899a0216012aee87d74b0f318085cd496beb46b0c5bc3e0b3213"} Nov 02 20:56:04.965210 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.965145 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp" Nov 02 20:56:04.974554 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:04.974541 5420 scope.go:122] "RemoveContainer" containerID="56dcd9f299e63dc3eb565ab28c7f2214099de50023d801984df20c8e062604fc" Nov 02 20:56:05.118051 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:05.118030 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp"] Nov 02 20:56:05.165731 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:05.165705 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-zd2pp"] Nov 02 20:56:05.224137 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:05.224115 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g"] Nov 02 20:56:05.228208 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:05.228191 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-qfm4g"] Nov 02 20:56:05.773222 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:05.773195 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" path="/var/lib/kubelet/pods/69a9c98d-5ec6-4e6d-9178-6bb06f51888e/volumes" Nov 02 20:56:05.773608 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:56:05.773595 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" path="/var/lib/kubelet/pods/e44f81ce-47d4-4b30-ab50-90135d0b45d2/volumes" Nov 02 20:56:32.657193 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:56:32.657157 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:57:02.656921 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:57:02.656882 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:57:28.280096 ip-10-0-134-206 kubenswrapper[5420]: I1102 20:57:28.280055 5420 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal" podUID="d51646e0acf14788341d5bd3c94d4017" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 20:57:32.656968 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:57:32.656932 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:58:02.658390 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:58:02.658301 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:58:32.656385 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:58:32.656345 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:59:02.657519 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:59:02.657483 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 20:59:32.656851 ip-10-0-134-206 kubenswrapper[5420]: E1102 20:59:32.656720 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:00:00.173147 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173115 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv"] Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173473 5420 cpu_manager.go:414] "containerMap: removing container" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173483 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173497 5420 cpu_manager.go:414] "containerMap: removing container" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173502 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173516 5420 cpu_manager.go:414] "containerMap: removing container" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173521 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173536 5420 cpu_manager.go:414] "containerMap: removing container" podUID="91a3c81d-7baa-4a47-b018-931d8e75c077" containerName="discovery" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173540 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91a3c81d-7baa-4a47-b018-931d8e75c077" containerName="discovery" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173601 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b79a65-9be4-4acb-b68b-19dca7de84cd" containerName="istio-proxy" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173610 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44f81ce-47d4-4b30-ab50-90135d0b45d2" containerName="istio-proxy" Nov 02 21:00:00.173616 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173618 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a9c98d-5ec6-4e6d-9178-6bb06f51888e" containerName="istio-proxy" Nov 02 21:00:00.173960 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.173627 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="91a3c81d-7baa-4a47-b018-931d8e75c077" containerName="discovery" Nov 02 21:00:00.176812 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.176793 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:00.225758 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.225733 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 21:00:00.271764 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.271736 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 21:00:00.271974 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.271939 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9xknx\"" Nov 02 21:00:00.273765 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.273744 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv"] Nov 02 21:00:00.312198 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.312172 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjcc\" (UniqueName: \"kubernetes.io/projected/4fbdf9cd-62c6-476e-89d6-82241d0fb774-kube-api-access-pqjcc\") pod \"maas-api-key-cleanup-29894220-zxllv\" (UID: \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:00.312293 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.312242 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbdf9cd-62c6-476e-89d6-82241d0fb774-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-zxllv\" (UID: \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:00.412817 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.412798 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjcc\" (UniqueName: \"kubernetes.io/projected/4fbdf9cd-62c6-476e-89d6-82241d0fb774-kube-api-access-pqjcc\") pod \"maas-api-key-cleanup-29894220-zxllv\" (UID: \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:00.412907 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.412845 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbdf9cd-62c6-476e-89d6-82241d0fb774-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-zxllv\" (UID: \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:00.413391 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.413374 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbdf9cd-62c6-476e-89d6-82241d0fb774-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894220-zxllv\" (UID: \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:00.470356 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.470312 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjcc\" (UniqueName: \"kubernetes.io/projected/4fbdf9cd-62c6-476e-89d6-82241d0fb774-kube-api-access-pqjcc\") pod \"maas-api-key-cleanup-29894220-zxllv\" (UID: \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:00.488584 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.488558 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:00.850589 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:00.850565 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv"] Nov 02 21:00:00.852370 ip-10-0-134-206 kubenswrapper[5420]: W1102 21:00:00.852343 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbdf9cd_62c6_476e_89d6_82241d0fb774.slice/crio-897652ef0e28650d232e5afa39155d717fa6e9cd23b0c79c6e4dbbe03c96776c WatchSource:0}: Error finding container 897652ef0e28650d232e5afa39155d717fa6e9cd23b0c79c6e4dbbe03c96776c: Status 404 returned error can't find the container with id 897652ef0e28650d232e5afa39155d717fa6e9cd23b0c79c6e4dbbe03c96776c Nov 02 21:00:01.122064 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:01.122007 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" event={"ID":"4fbdf9cd-62c6-476e-89d6-82241d0fb774","Type":"ContainerStarted","Data":"f7ad3e774266a9c8cb7f62a81471453b454cb6fd3f9814e1d3938426a5d68f9d"} Nov 02 21:00:01.122064 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:01.122055 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" event={"ID":"4fbdf9cd-62c6-476e-89d6-82241d0fb774","Type":"ContainerStarted","Data":"897652ef0e28650d232e5afa39155d717fa6e9cd23b0c79c6e4dbbe03c96776c"} Nov 02 21:00:01.231633 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:01.231586 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" podStartSLOduration=1.2315737979999999 podStartE2EDuration="1.231573798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:00:01.230064799 +0000 UTC m=+2673.594676005" watchObservedRunningTime="2026-11-02 21:00:01.231573798 +0000 UTC m=+2673.596185003" Nov 02 21:00:02.129129 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:02.129100 5420 generic.go:356] "Generic (PLEG): container finished" podID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerID="f7ad3e774266a9c8cb7f62a81471453b454cb6fd3f9814e1d3938426a5d68f9d" exitCode=60 Nov 02 21:00:02.129286 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:02.129183 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" event={"ID":"4fbdf9cd-62c6-476e-89d6-82241d0fb774","Type":"ContainerDied","Data":"f7ad3e774266a9c8cb7f62a81471453b454cb6fd3f9814e1d3938426a5d68f9d"} Nov 02 21:00:02.129392 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:02.129376 5420 scope.go:122] "RemoveContainer" containerID="f7ad3e774266a9c8cb7f62a81471453b454cb6fd3f9814e1d3938426a5d68f9d" Nov 02 21:00:02.656810 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:00:02.656741 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:00:03.135998 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:03.135970 5420 generic.go:356] "Generic (PLEG): container finished" podID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerID="434fda0b720a38c2888c9012e668d903370e89f1ffceac541014152985b410c4" exitCode=60 Nov 02 21:00:03.136152 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:03.136052 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" event={"ID":"4fbdf9cd-62c6-476e-89d6-82241d0fb774","Type":"ContainerDied","Data":"434fda0b720a38c2888c9012e668d903370e89f1ffceac541014152985b410c4"} Nov 02 21:00:03.136152 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:03.136108 5420 scope.go:122] "RemoveContainer" containerID="f7ad3e774266a9c8cb7f62a81471453b454cb6fd3f9814e1d3938426a5d68f9d" Nov 02 21:00:03.136272 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:03.136257 5420 scope.go:122] "RemoveContainer" containerID="434fda0b720a38c2888c9012e668d903370e89f1ffceac541014152985b410c4" Nov 02 21:00:03.136450 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:00:03.136429 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-zxllv_odh-ai-gateway-infra(4fbdf9cd-62c6-476e-89d6-82241d0fb774)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" Nov 02 21:00:04.146282 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:04.146254 5420 scope.go:122] "RemoveContainer" containerID="434fda0b720a38c2888c9012e668d903370e89f1ffceac541014152985b410c4" Nov 02 21:00:04.146638 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:00:04.146464 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-zxllv_odh-ai-gateway-infra(4fbdf9cd-62c6-476e-89d6-82241d0fb774)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" Nov 02 21:00:16.767514 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:16.767476 5420 scope.go:122] "RemoveContainer" containerID="434fda0b720a38c2888c9012e668d903370e89f1ffceac541014152985b410c4" Nov 02 21:00:17.212584 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:17.212556 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" event={"ID":"4fbdf9cd-62c6-476e-89d6-82241d0fb774","Type":"ContainerStarted","Data":"189e981e55cb65dfeedbea0bbbe5b6103df16624ed6da44c8e38457b92aadc17"} Nov 02 21:00:18.219710 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:18.219655 5420 generic.go:356] "Generic (PLEG): container finished" podID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerID="189e981e55cb65dfeedbea0bbbe5b6103df16624ed6da44c8e38457b92aadc17" exitCode=60 Nov 02 21:00:18.220179 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:18.219707 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" event={"ID":"4fbdf9cd-62c6-476e-89d6-82241d0fb774","Type":"ContainerDied","Data":"189e981e55cb65dfeedbea0bbbe5b6103df16624ed6da44c8e38457b92aadc17"} Nov 02 21:00:18.220179 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:18.219760 5420 scope.go:122] "RemoveContainer" containerID="434fda0b720a38c2888c9012e668d903370e89f1ffceac541014152985b410c4" Nov 02 21:00:18.220179 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:18.220027 5420 scope.go:122] "RemoveContainer" containerID="189e981e55cb65dfeedbea0bbbe5b6103df16624ed6da44c8e38457b92aadc17" Nov 02 21:00:18.220344 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:00:18.220229 5420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894220-zxllv_odh-ai-gateway-infra(4fbdf9cd-62c6-476e-89d6-82241d0fb774)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" Nov 02 21:00:18.335776 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:18.335747 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv"] Nov 02 21:00:19.351955 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:19.351933 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:19.444813 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:19.444791 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbdf9cd-62c6-476e-89d6-82241d0fb774-kube-api-access-pqjcc\" (UniqueName: \"kubernetes.io/projected/4fbdf9cd-62c6-476e-89d6-82241d0fb774-kube-api-access-pqjcc\") pod \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\" (UID: \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\") " Nov 02 21:00:19.444936 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:19.444859 5420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbdf9cd-62c6-476e-89d6-82241d0fb774-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbdf9cd-62c6-476e-89d6-82241d0fb774-maas-api-service-ca\") pod \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\" (UID: \"4fbdf9cd-62c6-476e-89d6-82241d0fb774\") " Nov 02 21:00:19.445216 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:19.445195 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbdf9cd-62c6-476e-89d6-82241d0fb774-maas-api-service-ca" pod "4fbdf9cd-62c6-476e-89d6-82241d0fb774" (UID: "4fbdf9cd-62c6-476e-89d6-82241d0fb774"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 21:00:19.446796 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:19.446776 5420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbdf9cd-62c6-476e-89d6-82241d0fb774-kube-api-access-pqjcc" pod "4fbdf9cd-62c6-476e-89d6-82241d0fb774" (UID: "4fbdf9cd-62c6-476e-89d6-82241d0fb774"). InnerVolumeSpecName "kube-api-access-pqjcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 21:00:19.545429 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:19.545356 5420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjcc\" (UniqueName: \"kubernetes.io/projected/4fbdf9cd-62c6-476e-89d6-82241d0fb774-kube-api-access-pqjcc\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:19.545429 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:19.545383 5420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fbdf9cd-62c6-476e-89d6-82241d0fb774-maas-api-service-ca\") on node \"ip-10-0-134-206.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 21:00:20.232977 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:20.232944 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" event={"ID":"4fbdf9cd-62c6-476e-89d6-82241d0fb774","Type":"ContainerDied","Data":"897652ef0e28650d232e5afa39155d717fa6e9cd23b0c79c6e4dbbe03c96776c"} Nov 02 21:00:20.233136 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:20.233019 5420 scope.go:122] "RemoveContainer" containerID="189e981e55cb65dfeedbea0bbbe5b6103df16624ed6da44c8e38457b92aadc17" Nov 02 21:00:20.233136 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:20.232955 5420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv" Nov 02 21:00:20.386502 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:20.386478 5420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv"] Nov 02 21:00:20.432469 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:20.432441 5420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894220-zxllv"] Nov 02 21:00:21.774567 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:00:21.774527 5420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" path="/var/lib/kubelet/pods/4fbdf9cd-62c6-476e-89d6-82241d0fb774/volumes" Nov 02 21:00:32.657640 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:00:32.657599 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:01:02.656553 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:01:02.656463 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:01:32.657296 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:01:32.657260 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:02:02.656739 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:02:02.656698 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:02:32.659007 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:02:32.658918 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:02:53.979071 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:02:53.979037 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-86f597b94c-588d2_76f30471-010d-4c78-8d5c-7b1fef0f956f/manager/0.log" Nov 02 21:02:54.577802 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:02:54.577778 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-d8fff_cf565282-2490-4e6a-af4b-d85efbf5bab3/manager/0.log" Nov 02 21:02:55.456195 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:02:55.456168 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-ntg25_1389a165-be88-48ce-abd7-17480feb6620/server/0.log" Nov 02 21:03:00.406278 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:00.406248 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2r66m_7d490bf4-b010-4edc-b4b6-a65fc0e2087d/manager/0.log" Nov 02 21:03:00.865297 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:00.865270 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xr5ql_06ff9d5a-cfc4-4115-9ed8-e595ee874bd2/manager/0.log" Nov 02 21:03:02.528711 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:02.528663 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-676d5b6d89-lrjtz_49e58208-6c5c-44d3-9750-42b6884a3325/kube-auth-proxy/0.log" Nov 02 21:03:02.657024 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:03:02.656992 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:03:02.761081 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:02.761059 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-8vqkm_eccd3c31-d213-4763-b79b-ee970b2c334d/istio-proxy/1.log" Nov 02 21:03:03.224646 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:03.224623 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-bmkzw_eba2df43-5fb9-4e11-9bea-b7c793484821/payload-pre-processing/0.log" Nov 02 21:03:03.437482 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:03.437458 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6g7tz_b7165f85-4e19-4e3a-b333-cffb9d791cf6/payload-processing/0.log" Nov 02 21:03:04.129158 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:04.129127 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-8vqkm_eccd3c31-d213-4763-b79b-ee970b2c334d/istio-proxy/0.log" Nov 02 21:03:04.941990 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:04.941961 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-z8sv4_d276f839-35ce-41a4-aa67-f46d4d405f3f/main/0.log" Nov 02 21:03:05.625252 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:05.625228 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8_3d09f5c8-eea4-44f5-8c9b-3643e053f0eb/main/0.log" Nov 02 21:03:05.834801 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:05.834781 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-x5gcv_fea054c8-a620-453c-862a-8ce7eeed9e0c/main/0.log" Nov 02 21:03:06.048766 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:06.048748 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj_774962ca-647f-40c4-855b-157ab23abc51/main/0.log" Nov 02 21:03:16.675263 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:16.675229 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-86f597b94c-588d2_76f30471-010d-4c78-8d5c-7b1fef0f956f/manager/0.log" Nov 02 21:03:17.259758 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:17.259729 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-d8fff_cf565282-2490-4e6a-af4b-d85efbf5bab3/manager/0.log" Nov 02 21:03:17.940875 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:17.940847 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-ntg25_1389a165-be88-48ce-abd7-17480feb6620/server/0.log" Nov 02 21:03:31.125196 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:31.125167 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2r66m_7d490bf4-b010-4edc-b4b6-a65fc0e2087d/manager/0.log" Nov 02 21:03:31.784768 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:31.784735 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xr5ql_06ff9d5a-cfc4-4115-9ed8-e595ee874bd2/manager/0.log" Nov 02 21:03:32.657136 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:03:32.657095 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:03:33.198111 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:33.198087 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-676d5b6d89-lrjtz_49e58208-6c5c-44d3-9750-42b6884a3325/kube-auth-proxy/0.log" Nov 02 21:03:33.414318 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:33.414293 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-8vqkm_eccd3c31-d213-4763-b79b-ee970b2c334d/istio-proxy/1.log" Nov 02 21:03:33.856090 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:33.856067 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-bmkzw_eba2df43-5fb9-4e11-9bea-b7c793484821/payload-pre-processing/0.log" Nov 02 21:03:34.084453 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:34.084430 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6g7tz_b7165f85-4e19-4e3a-b333-cffb9d791cf6/payload-processing/0.log" Nov 02 21:03:35.141598 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:35.141565 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-z8sv4_d276f839-35ce-41a4-aa67-f46d4d405f3f/main/0.log" Nov 02 21:03:35.806447 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:35.806423 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82fws8_3d09f5c8-eea4-44f5-8c9b-3643e053f0eb/main/0.log" Nov 02 21:03:36.037972 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:36.037949 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-x5gcv_fea054c8-a620-453c-862a-8ce7eeed9e0c/main/0.log" Nov 02 21:03:36.262474 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:03:36.262416 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-lq8hj_774962ca-647f-40c4-855b-157ab23abc51/main/0.log" Nov 02 21:04:02.657190 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:04:02.657109 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:04:24.039070 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:04:24.039032 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6g7tz_b7165f85-4e19-4e3a-b333-cffb9d791cf6/payload-processing/0.log" Nov 02 21:04:24.267843 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:04:24.267814 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6g7tz_b7165f85-4e19-4e3a-b333-cffb9d791cf6/payload-processing/0.log" Nov 02 21:04:32.658506 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:04:32.658472 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:05:02.656942 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:05:02.656907 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:05:10.081962 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:10.081897 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6g7tz_b7165f85-4e19-4e3a-b333-cffb9d791cf6/payload-processing/0.log" Nov 02 21:05:10.300236 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:10.300214 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-6g7tz_b7165f85-4e19-4e3a-b333-cffb9d791cf6/payload-processing/0.log" Nov 02 21:05:19.873310 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:19.873282 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s49d6_6b02c000-6d43-4adf-ba8b-4afd3e1186bc/global-pull-secret-syncer/0.log" Nov 02 21:05:23.255799 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:23.255767 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wnw5s_0ae28258-51a5-4e33-8040-39d8592006b8/konnectivity-agent/0.log" Nov 02 21:05:24.037224 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:24.037194 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-206.us-west-2.compute.internal_d51646e0acf14788341d5bd3c94d4017/haproxy/0.log" Nov 02 21:05:29.871262 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.871230 5420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz"] Nov 02 21:05:29.871645 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.871585 5420 cpu_manager.go:414] "containerMap: removing container" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerName="cleanup" Nov 02 21:05:29.871645 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.871595 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerName="cleanup" Nov 02 21:05:29.871645 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.871609 5420 cpu_manager.go:414] "containerMap: removing container" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerName="cleanup" Nov 02 21:05:29.871645 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.871613 5420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerName="cleanup" Nov 02 21:05:29.871806 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.871722 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerName="cleanup" Nov 02 21:05:29.871806 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.871735 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerName="cleanup" Nov 02 21:05:29.871806 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.871739 5420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbdf9cd-62c6-476e-89d6-82241d0fb774" containerName="cleanup" Nov 02 21:05:29.874801 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.874781 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:29.920808 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.920783 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-sys\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:29.920900 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.920812 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-lib-modules\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:29.920900 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.920835 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-podres\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:29.920975 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.920917 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-proc\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:29.920975 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.920960 5420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r2fg\" (UniqueName: \"kubernetes.io/projected/6452af11-8074-4a3d-b7b6-b9560102147e-kube-api-access-7r2fg\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:29.969740 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.969711 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p7vqs\"/\"openshift-service-ca.crt\"" Nov 02 21:05:29.969845 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.969778 5420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p7vqs\"/\"kube-root-ca.crt\"" Nov 02 21:05:29.969845 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.969814 5420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p7vqs\"/\"default-dockercfg-rkz88\"" Nov 02 21:05:29.971629 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:29.971610 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz"] Nov 02 21:05:30.021247 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021227 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-sys\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.021343 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021252 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-lib-modules\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.021343 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021276 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-podres\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.021343 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021311 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-proc\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.021343 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021326 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-sys\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.021534 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021361 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-lib-modules\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.021534 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021381 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-podres\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.021534 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021390 5420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r2fg\" (UniqueName: \"kubernetes.io/projected/6452af11-8074-4a3d-b7b6-b9560102147e-kube-api-access-7r2fg\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.021534 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.021404 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6452af11-8074-4a3d-b7b6-b9560102147e-proc\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.078275 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.078247 5420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r2fg\" (UniqueName: \"kubernetes.io/projected/6452af11-8074-4a3d-b7b6-b9560102147e-kube-api-access-7r2fg\") pod \"perf-node-gather-daemonset-xmmkz\" (UID: \"6452af11-8074-4a3d-b7b6-b9560102147e\") " pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.186936 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.186888 5420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.549388 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.549362 5420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz"] Nov 02 21:05:30.551172 ip-10-0-134-206 kubenswrapper[5420]: W1102 21:05:30.551143 5420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6452af11_8074_4a3d_b7b6_b9560102147e.slice/crio-edffec9f0124174cca22aa93888ec02cbdd2e225a2cca90b66820773e57ff53e WatchSource:0}: Error finding container edffec9f0124174cca22aa93888ec02cbdd2e225a2cca90b66820773e57ff53e: Status 404 returned error can't find the container with id edffec9f0124174cca22aa93888ec02cbdd2e225a2cca90b66820773e57ff53e Nov 02 21:05:30.670764 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.670741 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2r66m_7d490bf4-b010-4edc-b4b6-a65fc0e2087d/manager/0.log" Nov 02 21:05:30.751976 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.751952 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" event={"ID":"6452af11-8074-4a3d-b7b6-b9560102147e","Type":"ContainerStarted","Data":"06fdbf866d43371428f9bed90df927b747f0e411cb8c967a545bd1147ae84a9b"} Nov 02 21:05:30.752087 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.752018 5420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" event={"ID":"6452af11-8074-4a3d-b7b6-b9560102147e","Type":"ContainerStarted","Data":"edffec9f0124174cca22aa93888ec02cbdd2e225a2cca90b66820773e57ff53e"} Nov 02 21:05:30.752087 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.752038 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:30.860360 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:30.860289 5420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" podStartSLOduration=1.860276872 podStartE2EDuration="1.860276872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 21:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 21:05:30.858978639 +0000 UTC m=+3003.223589843" watchObservedRunningTime="2026-11-02 21:05:30.860276872 +0000 UTC m=+3003.224888119" Nov 02 21:05:31.388267 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:31.388245 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-xr5ql_06ff9d5a-cfc4-4115-9ed8-e595ee874bd2/manager/0.log" Nov 02 21:05:32.657521 ip-10-0-134-206 kubenswrapper[5420]: E1102 21:05:32.657486 5420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x5lv8" containerName="extractor" Nov 02 21:05:36.766682 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:36.766638 5420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p7vqs/perf-node-gather-daemonset-xmmkz" Nov 02 21:05:37.030157 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:37.030068 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g29mn_bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/kube-state-metrics/0.log" Nov 02 21:05:37.417116 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:37.417086 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g29mn_bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/kube-rbac-proxy-main/0.log" Nov 02 21:05:37.808960 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:37.808904 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-g29mn_bb7c610f-2bc8-45db-b648-3d7b4d3cee2b/kube-rbac-proxy-self/0.log" Nov 02 21:05:38.612759 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:38.612735 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-ctj2p_3824767d-0c5a-420e-9541-0ca551ee1e24/monitoring-plugin/0.log" Nov 02 21:05:39.009563 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:39.009504 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tlrn_aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/node-exporter/0.log" Nov 02 21:05:39.399456 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:39.399433 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tlrn_aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/kube-rbac-proxy/0.log" Nov 02 21:05:39.789860 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:39.789804 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tlrn_aa0a953f-8e98-4fd0-aaeb-0a450e0c1ac3/init-textfile/0.log" Nov 02 21:05:42.590939 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:42.590913 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dfdh_d804b66e-c2da-42b6-add0-c11761995171/kube-rbac-proxy-main/0.log" Nov 02 21:05:42.985382 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:42.985315 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dfdh_d804b66e-c2da-42b6-add0-c11761995171/kube-rbac-proxy-self/0.log" Nov 02 21:05:43.378277 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:43.378253 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dfdh_d804b66e-c2da-42b6-add0-c11761995171/openshift-state-metrics/0.log" Nov 02 21:05:43.786398 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:43.786336 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c193ef7-627a-4721-988c-2c72e7473b1c/prometheus/0.log" Nov 02 21:05:44.183020 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:44.182998 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c193ef7-627a-4721-988c-2c72e7473b1c/config-reloader/0.log" Nov 02 21:05:44.574435 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:44.574414 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c193ef7-627a-4721-988c-2c72e7473b1c/thanos-sidecar/0.log" Nov 02 21:05:44.975893 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:44.975826 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c193ef7-627a-4721-988c-2c72e7473b1c/kube-rbac-proxy-web/0.log" Nov 02 21:05:45.368085 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:45.368061 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c193ef7-627a-4721-988c-2c72e7473b1c/kube-rbac-proxy/0.log" Nov 02 21:05:45.759344 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:45.759292 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c193ef7-627a-4721-988c-2c72e7473b1c/kube-rbac-proxy-thanos/0.log" Nov 02 21:05:46.152337 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:46.152318 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0c193ef7-627a-4721-988c-2c72e7473b1c/init-config-reloader/0.log" Nov 02 21:05:48.948864 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:48.948837 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb5949b9c-jjt6t_74280e02-7f3b-4a07-8bac-914610c5d04d/thanos-query/0.log" Nov 02 21:05:49.339313 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:49.339292 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb5949b9c-jjt6t_74280e02-7f3b-4a07-8bac-914610c5d04d/kube-rbac-proxy-web/0.log" Nov 02 21:05:49.731195 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:49.731125 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb5949b9c-jjt6t_74280e02-7f3b-4a07-8bac-914610c5d04d/kube-rbac-proxy/0.log" Nov 02 21:05:50.123229 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:50.123203 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb5949b9c-jjt6t_74280e02-7f3b-4a07-8bac-914610c5d04d/prom-label-proxy/0.log" Nov 02 21:05:50.515128 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:50.515069 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb5949b9c-jjt6t_74280e02-7f3b-4a07-8bac-914610c5d04d/kube-rbac-proxy-rules/0.log" Nov 02 21:05:50.905331 ip-10-0-134-206 kubenswrapper[5420]: I1102 21:05:50.905306 5420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb5949b9c-jjt6t_74280e02-7f3b-4a07-8bac-914610c5d04d/kube-rbac-proxy-metrics/0.log"